Predictive Factors of Research Productivity among Ophthalmology Residents: A Benchmark Analysis

Abstract

Introduction Positive and negative associations between prior publications and future research productivity is described in other fields, but no such analysis exists for ophthalmology. We conducted a study to determine characteristics of residents exhibiting research productivity during residency.

Methods Using San Francisco Match and Program Web sites, a roster of ophthalmology residents in 2019 to 2020 was compiled, and publication data was collected via PubMed and Google Scholar on a random sample of 100 third-year residents.

Results The median number of publications generated by ophthalmology residents before residency is 2 (range 0–13). Thirty-seven, 23, and 40 residents had zero, one, and two or more papers published during residency, respectively, with a median of 1 (range 0–14). On univariate analysis, compared with residents who published zero or one paper, those who published ≥ 2 were more likely to have more preresidency publications (odds ratio [OR] 1.30; p = 0.005), attend a top-25 ranked residency program by multiple metrics including Doximity reputation (OR 4.92; p < 0.001), and have attended a top-25 ranked medical school program by U.S. News and World Report (OR 3.24; p = 0.03). However, on adjusted analyses, the only factor that remained significant for predicting publications in residency was whether the residency program attended was top 25 ranked (OR 3.54; p = 0.009).

Discussion/Conclusion With the advent of the United States Medical Licensing Examination Step 1 pass/fail system, greater emphasis will be placed on other metrics, including research. This is the first benchmark analysis examining factors predictive of publication productivity in ophthalmology residents. Our study suggests that the residency program attended, not the medical school attended or prior publication history, plays an influential role in the number of publications produced during residency, highlighting the importance of factors to support research on the institutional level, such as mentorship and funding, rather than historical factors in research productivity by the resident.

Keywords

medical school - residency - research productivity - publications

Introduction

Ophthalmology continues to rank as one of the most competitive specialties with limited residency positions. According to the San Francisco (SF) Match Ophthalmology Residency Match Summary Report in 2020, only 495 applicants matched out of a total of 635 applicants (including international medical graduates) who participated, which correlates to a 78% match rate; the match rate for international medical graduates is significantly lower, ranging from 27 to 48% in the past few years.[1] In addition, United States Medical Licensing Examination (USMLE) Step 1 scores continue to increase for matched applicants, with an average of 245 in 2020, compared with 237 in 2011.[1] Given the competitiveness of the match process, applicants apply widely, averaging 76.4 programs, and spend around $6,000 per cycle.[2]

On February 12, 2020, the decision was made by the National Board of Medical Examiners and Federation of State Medical Boards to change the scoring system of the USMLE from a numerical score to pass/fail by 2022 at the earliest.[3] [4] Traditionally, higher USMLE scores have been a statistically significant predictor of matching both for U.S. graduates and international medical graduates.[5] [6] [7] With the recent outbreak of coronavirus disease 2019 (COVID-19) and the reduction, and in many cases, temporary elimination of visiting student rotations and in-person interviews,[8] the available metrics to gauge residency candidates has further dwindled. Although the full implications of these changes have not yet been realized, it is inevitable that program directors will begin to further scrutinize other metrics during the application process.

Research publications have been a quantifiable metric that have been used at multiple levels, from medical school to residency, fellowship, and faculty hiring. Publications may show the applicant's curiosity, drive to learn, and potential for pursuing research in the future; furthermore, research specifically in ophthalmology may demonstrate dedication toward the field. However, at this time it is unclear, in ophthalmology, the relationship between the matched applicant's preresidency publications and productivity during residency.

The purpose of this research study is to (1) establish recent quantitative baseline publication data of ophthalmology residents before and during residency, and to (2) characterize if bibliometrics such as the number, type, authorship placement, or other student-specific factors (degree, institution) have an association with the number of publications generated during residency.

Materials and Methods

Data Cohort

A list of ophthalmology residency programs was gathered using the SF Match residency program list and Association of University Professors of Ophthalmology list of programs. To capture as much information as possible during the residency years, postgraduate year 4 (PGY4) ophthalmology residents were selected. After a list of programs was created, each program Web site was searched to identify a list of their PGY4 residents, who comprised the initial pool of subjects. A simple random sample of 100 residents was then generated to create the cohort that was used in the study.

Web sites without available rosters or without indication of which residents made up which training year were excluded. In rare instances where rosters were not updated to reflect the 2019 to 2020 training year, the most recent roster available was used, and the date of the roster was noted to maintain parity in the final data analysis.

Information Collected

All information collected was publicly available on program Web sites or otherwise made available online. For residents, information collected included medical school, residency institution, year of graduation from medical school, degrees obtained (MD, DO, and/or PhD), and international medical graduate status.

Research Data Assessment

Scholarly productivity was assessed in two ways. First, a Hirsch index (h-index) was obtained for each individual. The h-index is a measure that takes into account the number and impact of publications measured by the number of published articles and the number of citations in peer-reviewed papers.[9] [10] The h-index was obtained by searching for the individual using the Scopus database. To determine if the author was a true match, the publications listed under the name had to either (1) contain a known academic institution affiliation to the author, or (2) be a publication related to the field of ophthalmology. In some instances of multiple entries on Scopus, the largest h-index value was used.

Second, a literature review was performed for each individual, which took place between March 2020 and July 2020. Similar criteria and methods to a study by Campbell et al[11] was used. Each individual was searched by first and last name on both Google Scholar and PubMed, using the search by author feature. If a maiden name was identified at some point during the search, both names were used for the study. In a few instances where the name was too common and hundreds of results were present, it was often not possible to complete an author search and so that individual was skipped and another one added to the list to maintain a total of 100 residents.

Research was credited to the author if it met at least one of two criteria: (1) the publication was in the field of ophthalmology in an appropriate time frame in relation to their training, or (2) the publication was associated with a known institutional affiliation at which the individual had attended. In addition, only original research, case reports, review articles, textbook chapters, and methods papers were included. All other papers, including conference abstracts, were excluded.

Multiple metrics were collected regarding each publication, including the title of the paper, the journal or textbook in which it was published, the associated journal impact factor and h-index, author order, and the type of paper (primary research, case report, textbook chapter, review article). To properly characterize at which time point a paper was published in a resident's training, the affiliated institution of the paper was noted and used to characterize whether a paper was published before residency or during residency. If an individual happened to attend the same institution for multiple levels of training, the year of publication was compared with the years they obtained each degree, and the publication was categorized according to when they graduated, inclusive of the graduation year.

Multiple ranking lists were used in the data analysis to categorize medical schools and residency programs. These included the 2019 National Institutes of Health research funding ranking for medical schools, 2021 U.S. News and World Report (USNWR) ranking for medical schools, and 2020 Doximity ophthalmology rankings by research output and reputation for residency programs.

Univariate analyses were conducted using logistic regression. Adjusted analyses were also done with logistic regression, and included three variables: number of preresidency publications, top 25 medical school attended by USNWR, and top 25 residency attended by Doximity reputation. Other metrics of ranking medical schools/residencies were not included in the logistic regression to reduce multicollinearity due to the similarity in institutional rankings across multiple metrics.

Results

The records of 454 PGY4 residents were initially included, and 100 of them were randomly selected, with 4 skipped and replaced because their names appeared too commonly during the literature search process. Of the 100 residents, 98 were PGY4 residents during the 2019 to 2020 academic year, while 2 of them were PGY4 residents during the 2018 to 2019 academic year. In total, 429 papers were identified as being authored by these residents. Note that 331 (77%) were ophthalmology-related papers (including papers published before and during residency), and 237 (55%) were on research conducted before residency began. [Table 1] displays a summary of the residents in the cohort as well as their publications. The mean number of publications before residency was 2.37 (standard deviation [SD] 2.63), with a median of 2 and range from 0 to 13. The mean number of publications during residency was 1.92 (SD 2.60), with a median of 1 and range from 0 to 14. Note that 57% of the articles were primary research articles, while 29% were case reports/case series, 11% were review papers, and 3% were textbook chapters.

Table 1 Summary of candidate predictors for 100 randomly selected ophthalmology residents in the United States

Feature	Count (%)
Degree	DO	3 (3.0)
	MD	90 (90.0)
	MD, PhD	7 (7.0)
Top 25 Medical School by USNWR	No	81 (81.0)
	Yes	19 (19.0)
Top 25 Medical School by NIH research funding	No	81 (81.0)
	Yes	19 (19.0)
Top 25 Residency Program by Doximity reputation	No	68 (68.0)
	Yes	32 (32.0)
Top 25 Residency Program by Doximity research output	No	71 (71.0)
	Yes	29 (29.0)
International medical graduate	No	100 (100.0)
	Yes	0 (0.0)
Last author in publication	No	97 (97.0)
	Yes	3 (3.0)
	Mean (Median, Range)
Average H-index of journals	93.6 (86.3, 0.0–737)
Average impact factor of journals	2.4 (2.3, 0.0–9.4)
Average author position in publications	2.2 (2.0, 1.0–8.0)
Individual H-index	1.9 (1.0, 0.0–10.0)
Total impactful contributions[a]	1.1 (1.0, 0.0–11.0)
Number of total ophthalmology-specific publications	3.3 (2.0, 0.0–19.0)
Number of preresidency publications	2.4 (2.0, 0.0–13.0)
Number of publications during residency	1.9 (1.0, 0.0–14.0)
Total publications	4.3 (3.0, 0.0–19.0)

Abbreviations: DO, Doctor of Osteopathic Medicine; H-index, Hirsch index; MD, Doctor of Medicine; NIH, National Institutes of Health; PhD, Doctor of Philosophy; USNWR, U.S. News and World Report.

^a Impactful contribution defined as either a first author, second author, or last author position on a publication.

Thirty-seven, 23, and 40 residents had zero, one, and two or more publications in residency, respectively. For the purposes of analysis, residents were grouped into either zero or one publication (n = 60), or two or more publications (n = 40) generated during residency. [Table 2] shows a summary of the univariate regression analysis conducted. Those who published two or more papers during residency had a significantly higher number of total publications in ophthalmology, with a mean of 6.30 (SD 3.87) whereas those who published one or less papers had a mean of 1.32 (SD 2.05) (odds ratio [OR] 1.96, p < 0.001). In addition, those who published two or more papers published in journals with higher impact factors with a mean 2.99 (SD 1.52) versus 1.98 (SD 1.89) (OR 1.40, p = 0.01) and published more often as the first, second, or last author with a mean of 1.90 (SD 2.41) papers versus 0.60 (SD 0.67) (OR 2.23, p = 0.003).

Table 2 Univariate regression analysis of residents with ≥ 2 publications during residency with the reference for all categorical predictors as the “no” condition

Feature	Odds ratio (95% CI)	p-Value
Top 25 Medical School by USNWR	3.24 (1.17–9.61)	0.03
Top 25 Medical School by NIH research funding	3.24 (1.17–9.61)	0.03
Top 25 Residency Program by Doximity reputation	4.92 (2.04–12.51)	< 0.001
Top 25 Residency Program by Doximity research output	3.64 (1.50–9.23)	0.005
Last author in publication	0.74 (0.03–8.02)	0.81
Average H-index of journals	1.00 (0.99–1.01)	0.20
Average impact factor of journals	1.40 (1.10–1.86)	0.01
Average author position in publications	1.01 (0.80–1.27)	0.93
Individual H-index	1.28 (1.05–1.63)	0.03
Total impactful contributions[a]	2.23 (1.42–4.18)	0.003
Number of total ophthalmology-specific publications	1.96 (1.55–2.63)	< 0.001
Number of preresidency publications	1.30 (1.10–1.59)	0.005

Abbreviations: CI, confidence interval; H-index, Hirsch index; NIH, National Institutes of Health; USNWR, U.S. News and World Report.

^a Impactful contribution defined as either a first author, second author, or last author position on a publication.

Note: Values <0.05, denoting significance, are in bold.

It was also found that those who published two or more papers in residency tended to have more publications before residency with a mean 3.35 (SD 2.92) versus 1.72 (SD 2.21) (OR 1.30, p = 0.005). In addition, those who attended a top 25 medical school by USNWR were more likely to publish more frequently before residency (OR 1.35, p = 0.002). They were also more likely to publish two or more papers in residency; of the 19 residents in the study that attended a top 25 medical school, 63% (12/19) had 2 or more papers in residency, compared with 35% (28/81) that did not attend one of these medical schools (OR 3.25, p = 0.03).

Those who published two or more papers in residency also tended to attend residency programs ranked in the top 25 on Doximity by both reputation and research output. By Doximity reputation, 32 residents attended top 25 institutions; 66% (21/32) of these residents published two or more papers in residency, versus 28% (19/68) who did not attend one of these institutions (OR 4.92, p < 0.001). Of note, of the 37 residents that had zero publications during residency, only 5% (2/37) attended a top 25 institution by Doximity reputation. Similarly, by Doximity research output, 62% (18/29) of residents attending a top 25 institution published two or more papers, versus 31% (22/71) (OR 3.64, p = 0.005).

On adjusted analyses, multiple logistic regression using a three-variable model with publications before residency, top 25 medical school attended by USNWR, and top 25 residency by Doximity reputation showed that only attending a top 25 residency program remained significant (OR 3.54, p = 0.009), while publications before residency (OR 1.17, p = 0.09) and medical school attended (OR 1.67, p = 0.40) were no longer statistically significant.

Discussion

Research and scholarly activity have long been used to evaluate applicants,[6] [12] and helps to gauge both an applicant's commitment and intellectual interest in the field. As research, to an extent, is a quantifiable data point, it is likely that it may become an increasingly important factor for residency applicants. In addition, research productivity is one of the three major functions of higher education, along with teaching and public service.[13] As such, research can be heavily weighted in the hiring and promotion of faculty in academic medicine.[14] [15] [16] [17] At the residency applicant level, research both helps programs choose applicants who more likely to pursue an academic career, and also helps applicants choose which programs would better suit their future career goals.[18] Indeed, it is a common goal to wish to select and ultimately recruit future residents who will continue to advance knowledge and teach the next generation of ophthalmologists through an academic career.

In other fields, such as dermatology, plastic surgery, and orthopaedics, research publication rate and quality has been studied to examine scholarship productivity at the medical student, residency, and fellowship level to study both its effect on matching and also as a predictor of future research productivity and subsequent career in academia.[11] [19] [20] [21] Previous studies have suggested that the true number of publications of applicants may often be lower than the reported numbers in their application resume when the literature is reviewed,[11] although ophthalmology residents may have a lower percentage of publication misrepresentation than average.[22]

Because research productivity is a quantifiable metric, it is generally helpful for applicants to have an accurate representation of how many publications on average are being generated by their peers to gauge their own competitiveness. In light of recent changes to USMLE Step 1, this topic is particularly relevant now, for both program directors and applicants, as the new pass/fail system has reduced the number of quantitative metrics available to help discern applicant competitiveness.

Beyond establishing accurate publication baseline data for ophthalmology residents before and during their residency training, our research shows that the primary predictor of residency publication productivity is attending a highly ranked program with the infrastructure and culture to produce said research. Inherently, one may initially attribute this to the selection process, where the top research-oriented institutions select residents who attended higher ranking medical schools and conducted more research before entering residency. However, while on univariate analysis all of these variables are significant for research productivity in residency, our study shows that in a multiple regression model, only the residency program attended remains significant.

This suggests that research productivity as a resident can be attributed more to the work environment rather than the resident's historical training and research involvement. While there may be several reasons for this, it is reasonable to postulate that funding opportunities, mentorship, laboratories, project availability, and a strong research culture play a much more significant impact in the ability to publish research.

One study, conducted in computer science examining over 200,000 publications and over 2,000 faculty, used a matched-pairs design to isolate the effects of prestige of the current work environment versus the prestige of a faculty's historical training environment.[23] The study concluded that a faculty's current work environment, rather than the selection effect or prestige of doctorate training, drove productivity; instead, training in a prestigious environment helped new faculty to secure work in a prestigious department, which led to future success, but this study indicated a limited role for the predictive value of past training on future success.[23] This suggests the work environment continues to play a predominant role regardless of the level of training, implying that the impact of environmental factors does not diminish as one progresses further in their career.

One of the limiting factors of this study is the inability to capture research papers after the data collection date, which ended roughly around the same time as the end of the residency training year. It is expected that several trainees will have publications that they have worked on during their training period that will go on to be published years into the future. Compared with a traditional year, the number of publications may also have further diminished at the onset of the COVID-19 pandemic, which occurred during the time frame of this study. In addition, because of the strict inclusion criteria for papers, a paper could not be included if the paper was outside of ophthalmology and it could not be verified that the affiliated institution was one that the resident had attended. This likely led to a more accurate representation of the number of papers published in ophthalmology, but may have underrepresented areas such as general medical education, quality improvement, and papers that were published at institutions outside of residents' training programs from undergraduate through residency.

Broadly, our study shows that residents attending research-oriented, prestigious institutions produce more research during residency, irrespective of their research productivity in medical school or their training background. This emphasizes the importance of the environmental factors present during residency and challenges the view that those who enter these prestigious institutions publish more research because of self-selective factors, which can be extrapolated from their past training and research. While top institutions undoubtedly tend to attract applications who have a passion for pursuing scholarly activity, and while self-selection does occur at multiple levels in training, this study suggests that scholarly activity in residency is determined by the program environment and not a resident's prior research activity or medical school. However, publication history and prior site of training do ultimately contribute to the institution that residents attend, and thus this may ultimately help secure their future productivity. It is also possible that, regardless of prior publications before residency, residents who wish to pursue a higher volume of research in residency self-select for certain institutions, or are selected for by these institutions based off of other factors, such as intent to pursue academia, that lead to higher productivity.

It is unclear whether productivity in residency or prestige of the residency program has any effect on future productivity in academia, or whether the working environment continues to be the dominant effect at the faculty level. Further studies are needed to examine these predictors and to determine whether publications during the training predict pursuit of fellowship/academic medicine and productivity in academia. It is also important to note that there are several factors outside of research that make residents strong assets to their programs and communities, and some trainees may self-select for programs with a lesser emphasis on a research component. In the new era of virtual interviews and changes in USMLE scoring, it will be important to examine other strategies beyond publication history to select among future applicants.

Conflict of Interest

None of the authors have any conflicts of interest to declare.

Acknowledgments

None.

• References

• 1 Report ORMS. . SF Match Residency and Fellowship Matching Services; 2020
• 2 Venincasa MJ, Cai LZ, Gedde SJ, Uhler T, Sridhar J. Current applicant perceptions of the ophthalmology residency match. JAMA Ophthalmol 2020; 138 (05) 460-466
• 3 Federation of State Medical Boards. USMLE Program Announces Upcoming Policy Changes. Published 2020. Accessed October 16, 2020 at: https://www.fsmb.org/advocacy/news-releases/usmle-program-announces-upcoming-policy-changes/
• 4 Chaudhry HJ, Katsufrakis PJ, Tallia AF. The USMLE Step 1 decision: an opportunity for medical education and training. JAMA 2020; 323 (20) 2017-2018
• 5 Loh AR, Joseph D, Keenan JD, Lietman TM, Naseri A. Predictors of matching in an ophthalmology residency program. Ophthalmology 2013; 120 (04) 865-870
• 6 Driver TH, Loh AR, Joseph D, Keenan JD, Naseri A. Predictors of matching in ophthalmology residency for international medical graduates. Ophthalmology 2014; 121 (04) 974-975.e2
• 7 Siatkowski RM, Mian SI, Culican SM. et al; Association of University Professors of Ophthalmology. Probability of success in the ophthalmology residency match: three-year outcomes analysis of San Francisco Matching Program Data. J Acad Ophthalmol 2018; 10 (01) e150-e157
• 8 Quillen DA, Siatkowski RM, Feldon S. of the Association of University Professors of Ophthalmology. COVID-19 and the ophthalmology match. Ophthalmology 2021; 128 (02) 181-184
• 9 Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005; 102 (46) 16569-16572
• 10 Hirsch JE. Does the H index have predictive power?. Proc Natl Acad Sci U S A 2007; 104 (49) 19193-19198
• 11 Campbell ST, Gupta R, Avedian RS. The effect of applicant publication volume on the orthopaedic residency match. J Surg Educ 2016; 73 (03) 490-495
• 12 Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology 2010; 117 (05) 1041-1047
• 13 Gilmore JL, To D-L. Evaluating academic productivity and quality. New Dir Institutional Res 1992; 1992 (75) 35-47
• 14 Atasoylu AA, Wright SM, Beasley BW. et al. Promotion criteria for clinician-educators. J Gen Intern Med 2003; 18 (09) 711-716
• 15 Beasley BW, Wright SM, Cofrancesco Jr J, Babbott SF, Thomas PA, Bass EB. Promotion criteria for clinician-educators in the United States and Canada. A survey of promotion committee chairpersons. JAMA 1997; 278 (09) 723-728
• 16 Dixon AK. Publishing and academic promotion. Singapore Med J 2009; 50 (09) 847-850
• 17 Todisco A, Souza RF, Gores GJ. Trains, tracks, and promotion in an academic medical center. Gastroenterology 2011; 141 (05) 1545-1548
• 18 Yousuf SJ, Kwagyan J, Jones LS. Applicants' choice of an ophthalmology residency program. Ophthalmology 2013; 120 (02) 423-427
• 19 Thompson RH, Lohse CM, Husmann DA, Leibovich BC, Gettman MT. Predictors of scholarly productivity, pursuit of fellowship, and academic practice among urology residents using medical student application materials. Urology 2018; 120: 49-55
• 20 Grewal NS, Spoon DB, Kawamoto HK. et al. Predictive factors in identifying subspecialty fellowship applicants who will have academic practices. Plast Reconstr Surg 2008; 122 (04) 1264-1271 , discussion 1272–1263
• 21 John AM, Gupta AB, John ES, Lopez SA, Lee B, Lambert WC. The impact of fellowship training on scholarly productivity in academic dermatology. Cutis 2016; 97 (05) 353-358
• 22 Wiggins MN. Misrepresentation by ophthalmology residency applicants. Arch Ophthalmol 2010; 128 (07) 906-910
• 23 Way SF, Morgan AC, Larremore DB, Clauset A. Productivity, prominence, and the effects of academic environment. Proc Natl Acad Sci U S A 2019; 116 (22) 10729-10733

Publikationsverlauf

Eingereicht: 31. August 2021

Angenommen: 16. März 2022

Artikel online veröffentlicht:
27. Juli 2022