Guiding Residency Program Educational Goals Using Institutional Keyword Reports from the Ophthalmic Knowledge Assessment Program Examination

Abstract

Introduction Residency programs receive an institutional keyword report following the annual Ophthalmic Knowledge Assessment Program (OKAP) examination containing the raw number of incorrectly answered questions. Programs would benefit from a method to compare relative performance between subspecialty sections. We propose a technique of normalizing the keyword report to determine relative subspecialty strengths and weaknesses in trainee performance.

Methods We retrospectively reviewed our institutional keyword reports from 2017 to 2019. We normalized the percentage of correctly answered questions for each postgraduate year (PGY) level by dividing the percent of correctly answered questions for each subspecialty by the percent correct across all subsections for that PGY level. We repeated this calculation for each PGY level in each subsection for each calendar year of analysis.

Results There was a statistically significant difference in mean performance between the subspecialty sections (p = 0.038). We found above average performance in the Uveitis and Ocular Inflammation section (95% confidence interval [CI]: 1.02–1.18) and high variability of performance in the Clinical Optics section (95% CI: 0.76–1.34).

Discussion The OKAP institutional keyword reports are extremely valuable for residency program self-evaluation. Performance normalized for PGY level and test year can reveal insightful trends into the relative strengths and weaknesses of trainee knowledge and guide data-driven curriculum improvement.

Keywords

Ophthalmic Knowledge Assessment Program - institution keyword reports - residency program - didactics - education - curriculum

Publication History

Received: 15 June 2020

Accepted: 31 August 2020

Article published online:
04 November 2020

• References

• 1 Liesegang TJ. New directions for the Ophthalmic Knowledge Assessment Program (OKAP) examination. Ophthalmology 1994; 101 (01) 194-198
• 2 O'Day DM. Examinations and ophthalmic education. Ophthalmology 2012; 119 (10) 1947-1948
• 3 Zafar S, Wang X, Srikumaran D. et al. Resident and program characteristics that impact performance on the Ophthalmic Knowledge Assessment Program (OKAP). BMC Med Educ 2019; 19 (01) 190
• 4 Chen PP. Resident OKAP performance. Ophthalmology 2012; 119 (03) 656
• 5 Rapuano C, Stout T. Ophthalmic Knowledge Assessment Program Committee. OKAP User's Guide. . Available at: https://www.aao.org/okap-exam. 2019. Accessed September 15, 2020
• 6 Oke I, Siegel NH, Peeler CE. et al. Completing the basic and clinical science course as a first-year ophthalmology resident. J Acad Ophthalmol 2019; 11: e54-e58
• 7 Arnold A. Developing the educational milestones for ophthalmology. J Grad Med Educ 2014; 6 (01) (Suppl. 01) 144-145
• 8 Lee AG, Oetting TA, Blomquist PH. et al. A multicenter analysis of the ophthalmic knowledge assessment program and American Board of Ophthalmology written qualifying examination performance. Ophthalmology 2012; 119 (10) 1949-1953
• 9 Carey A, Drucker M. Standardized Training Examinations among Ophthalmology Residents and the American Board of Ophthalmology Written Qualifying Examination First Attempt: The Morsani College of Medicine Experience. Journal of Academic Ophthalmology 2014; 07: e008-e012
• 10 Kempton JE, Shields MB, Afshari NA, Dou W, Adelman RA. Fellow selection criteria. Ophthalmology 2009; 116 (05) 1020-1020.e4
• 11 Johnson GA, Bloom JN, Szczotka-Flynn L, Zauner D, Tomsak RL. A comparative study of resident performance on standardized training examinations and the American Board of Ophthalmology Written Examination. Ophthalmology 2010; 117 (12) 2435-2439