Key Word Use in Letters of Recommendation for Ophthalmology Residency Applicants According to Race, Gender, and AchievementsFunding This study was supported in part by an unrestricted departmental grant from the Research to Prevent Blindness and the Heed Foundation to one of the authors (S. G.).
20 July 2018
11 October 2018
21 November 2018 (online)
Objectives To identify differences in letters of recommendation (LORs) of applicants to a single ophthalmology residency program by gender, race, academic performance, and match outcome.
Design This was a retrospective analysis of LORS for 2,523 applicants (7,569 letters) to the University of California, Irvine ophthalmology residency program from 2011 to 2018.
Methods Programming scripts were employed to determine the number of times 22 key words from four thematic categories (standout words, ability, grindstone, and compassion) appeared in LORs for each applicant. A chi-square test was performed to assess for possible differences in the presence of each key word by the following characteristics: gender, underrepresented minority (URM) status, Alpha Omega Alpha (AOA) membership, the United States Medical Licensing Exam (USMLE) Step 1 score, and match outcome. Linear regressions were created to determine the frequency at which words in each thematic category appeared according to the same baseline characteristics.
Results In the LORs, females were more likely to be described as “empathetic” (p = 0.002), URMs were more likely to be described as “caring” (p = 0.002), high Step 1 scorers (≥240) were more likely to be described as “outstanding” (p = 0.002), and matched students were more likely to be described as “exceptional” (p = 0.001), “outstanding” (p < 0.001), and “superb” (p = 0.001). Standout words appeared more often in the LORs of AOA members, matched candidates, and high Step 1 scorers (p < 0.001 for all comparisons). “Competent” appeared more commonly in LORs for low Step 1 scorers (p < 0.001) and unmatched applicants (p = 0.001).
Conclusion This study identifies differences in LORs by gender, URM status, and achievement including successful ophthalmology residency match. Females and URMs were more likely to be described as “empathetic” and “caring,” respectively; otherwise, we detected no gender or racial disparities in key word use in LORs. Candidates with high USMLE Step 1 scores or AOA membership had a higher frequency of standout words in their LORs. Whether they were truly more qualified in various dimensions or if they benefited from a halo effect bias warrants further investigation. There was a significant difference in the number of standout words in LORs between matched and unmatched applicants, suggesting that key word frequency may be a relevant metric for LOR appraisal.
- 1 SF Match. Ophthalmology Residency Match Summary Report 2018. Available at: https://www.sfmatch.org/PDFFilesDisplay/Ophthalmology_Residency_Stats_2018.pdf . Accessed June 15, 2018
- 2 Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology 2010; 117 (05) 1041-1047
- 3 Grubbs Jr JR, Mian SI. Advising students interested in ophthalmology: a summary of the evidence. Ophthalmology 2016; 123 (07) 1406-1410
- 4 Loh AR, Joseph D, Keenan JD, Lietman TM, Naseri A. Predictors of matching in an ophthalmology residency program. Ophthalmology 2013; 120 (04) 865-870
- 5 SF Match. Ophthalmology Residency Standardized Letter of Reference. Available at: https://www.sfmatch.org/PDFFilesDisplay/OPHTHR_StandardizedLOR.pdf . Accessed June 16, 2018
- 6 Lee AG, Golnik KC, Oetting TA. , et al. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol 2008; 53 (02) 164-176
- 7 Pololi LH, Civian JT, Brennan RT, Dottolo AL, Krupat E. Experiencing the culture of academic medicine: gender matters, a national study. J Gen Intern Med 2013; 28 (02) 201-207
- 8 Madera JM, Hebl MR, Martin RC. Gender and letters of recommendation for academia: agentic and communal differences. J Appl Psychol 2009; 94 (06) 1591-1599
- 9 Schmader T, Whitehead J, Wysocki VH. A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles 2007; 57 (7-8) 509-514
- 10 Trix F, Psenka C. Exploring the color of glass: letters of recommendation for female and male medical faculty. Discourse Soc 2003; 14 (02) 191-220
- 11 Ross DA, Boatright D, Nunez-Smith M, Jordan A, Chekroud A, Moore EZ. Differences in words used to describe racial and gender groups in Medical Student Performance Evaluations. PLoS ONE 2017; 12 (08) e0181659
- 12 Isaac C, Chertoff J, Lee B, Carnes M. Do students' and authors' genders affect evaluations? A linguistic analysis of Medical Student Performance Evaluations. Acad Med 2011; 86 (01) 59-66
- 13 Lachman SJ, Bass AR. A direct study of halo effect. J Psychol 1985; 119 (06) 535-540
- 14 Gibson JL, Gore JS. Is he a hero or a weirdo? How norm violations influence the halo effect. Gend Issues 2016; 33 (04) 299-310
- 15 Messner AH, Shimahara E. Letters of recommendation to an otolaryngology/head and neck surgery residency program: their function and the role of gender. Laryngoscope 2008; 118 (08) 1335-1344
- 16 Gauer JL, Jackson JB. Association between the Medical College Admission Test scores and Alpha Omega Alpha Medical Honors Society membership. Adv Med Educ Pract 2017; 8: 627-632
- 17 Cuddy MM, Swanson DB, Clauser BE. A multilevel analysis of examinee gender and USMLE step 1 performance. Acad Med 2008; 83 (10, Suppl): S58-S62
- 18 Case SM, Becker DF, Swanson DB. Performances of men and women on NBME Part I and Part II: the more things change…. Acad Med 1993; 68 (10, Suppl): S25-S27
- 19 Lypson ML, Ross PT, Hamstra SJ, Haftel HM, Gruppen LD, Colletti LM. Evidence for increasing diversity in graduate medical education: the competence of underrepresented minority residents measured by an intern objective structured clinical examination. J Grad Med Educ 2010; 2 (03) 354-359
- 20 Edmond MB, Deschenes JL, Eckler M, Wenzel RP. Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews. Acad Med 2001; 76 (12) 1253-1256
- 21 Tekian A, Han Y, Hruska L, Krainik AJ. Do underrepresented minority medical students differ from non-minority students in problem-solving ability?. Teach Learn Med 2001; 13 (02) 86-91
- 22 Xierali IM, Nivet MA, Wilson MR. Current and future status of diversity in ophthalmologist workforce. JAMA Ophthalmol 2016; 134 (09) 1016-1023
- 23 Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ 2007; 64 (06) 378-385
- 24 Stohl HE, Hueppchen NA, Bienstock JL. The utility of letters of recommendation in predicting resident success: can the ACGME competencies help?. J Grad Med Educ 2011; 3 (03) 387-390
- 25 Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg 2006; 202 (04) 649-654
- 26 Rim Y. How reliable are letters of recommendation?. J Higher Educ 1976; 47 (04) 437-445