U.S. News & World Report Ophthalmology Hospital Rankings and Research Productivity

Abstract

Introduction Despite the wide usage of U.S. News & World Report (U.S. News) rankings of ophthalmology hospitals among the public, residency applicants, and ophthalmologists, there is disagreement in the literature on the role of quality of care, research productivity, and other factors in the ranking system. This study investigated the association of U.S. News ranking of ophthalmology hospitals and objective measures of research productivity.

Methods The 2020 U.S. News “Best Hospitals for Ophthalmology” ranking lists 38 hospitals by reputation score and numerically ranks the top 12 institutions. For our analysis, top 12 hospitals were classified as group A and the remaining 26 as group B. The Clinicaltrials.gov, National Institutes of Health (NIH) Research Portfolio Online Reporting Tools Expenditures and Results (RePORTER), and NIH Research Portfolio Online Reporting Tools (RePORT) were systematically searched for total clinical trials, NIH funding, and the National Eye Institute (NEI) funding for fiscal years 2017, 2018, and 2019. Faculty size and the number of publications by ophthalmology faculty per hospital were recorded from a previous study in 2016.

Results Independent measures of research productivity significantly associated with group A status after multivariate logistic regression analysis were mean faculty Hirsch's index (h-index) over 15 (odds ratio [OR]: 6.13, 95% confidence interval [CI]: [1.14–32.94]) and conducting five or more total clinical trials (OR: 8.77, 95% CI: [1.39–55.16]).

Conclusion This study suggests that the reputation-based U.S. News ranking may serve as a proxy for an ophthalmology department's contribution to research measured by mean faculty h-index and number of clinical trials.

Keywords

U.S. News - ophthalmology - academic ranking - research productivity

The U.S. News & World Report (U.S. News) publishes a ranking for the top hospitals for ophthalmology each year. Based solely on opinion polls of ophthalmologists,[1] [2] the U.S. News ranking system is controversial but still widely used by the public, ophthalmologists, residency applicants, and researchers.[3] [4] [5] The U.S. News rankings of medical specialties may correlate stronger with reputation than objective measures,[6] [7] [8] though there is disagreement regarding the significance of quality of care, research productivity, and other factors in determining reputation.[9] [10]

To our knowledge, the U.S. News ranking system for ophthalmology has not been validated. However, academic hospitals have been shown to produce better hospital outcomes for many common and rare diseases.[11] [12] [13] [14] We investigated the association between U.S. News ranking for U.S. ophthalmology hospitals and research productivity.

Methods

The Brown University Institutional Review Board determined that this study did not meet the definition of human participants' research and did not require formal review.

Data Collection

The 2020 U.S. News “Best Hospitals for Ophthalmology” ranking lists 38 hospitals by reputation score, calculated by averaging 3 years (2017, 2018, and 2019) of survey responses from a sample of 12,838 board-certified ophthalmologists from the Doximity Masterfile.[1] Ophthalmology hospitals recommended by at least 5% of survey respondents are numerically ranked; for the 2020 ranking, 12 ophthalmology hospitals received a numerical ranking. Top 12 hospitals were classified as group A and the remaining 26 as group B. Any other ophthalmology hospitals not listed by U.S. News were not included in this study.

We used four validated objective measures of research output for each ophthalmology department's primary hospital affiliate as follows: (1) faculty publications, (2) clinical trials, (3) the National Eye Institute (NEI) grants, and (4) the National Institutes of Health (NIH) grants.[10] [15] In one case, two departments considered the same hospital as their primary affiliate; their total measures of research productivity were summed and mean faculty publications recalculated. In another case, a hospital was not considered a primary affiliate by any medical school, so the hospital was independently searched in each database.

The first measure of research productivity was the institution's faculty Hirsch index (h-index).[16] The summed and mean h-index and number of faculty members of each department were recorded from Thiessen et al.[17] The h-index is a standard measure for publication quality and productivity in health science research.[18] Given that an h-index of 20 is considered to represent a successful scientist after 20 years of research[18] and that most programs are staffed with a mixture of young and old faculty members; a mean faculty h-index of greater than 15 was defined as high research productivity.

The number of completed and ongoing clinical trials was recorded through a search of clinicaltrials.gov. Searches were filtered with the term “eye” and a start date in fiscal years 2017, 2018, and 2019. The number of clinical trials reflects the quantity of clinical research.

NIH funding per department was based on a report of NIH funding awarded to U.S. medical schools in fiscal years 2017, 2018, and 2019 published by NIH Research Portfolio Online Reporting Tools Expenditures and Results (RePORTER) and analyzed by Blue Ridge Medical Institute.[19] NIH funding is a validated measure of research productivity in ophthalmology.[20]

NEI research grant data per hospital in fiscal years 2017, 2018, and 2019 were obtained through NIH Research Portfolio Online Reporting Tools (RePORT), including basic science, translational, and clinical research. We used NEI funding data in addition to NIH data because many hospitals did not report to the NIH which department was receiving the funds, resulting in missing NIH data for many high-ranked departments. Additionally, different departments have different classifications for their faculty members; some medical schools may appear to have less funding because they do not classify basic science researchers as part of their ophthalmology departments. NEI funding includes all types of research, providing a uniform standard across all departments.

Statistical Analysis

Statistical analysis was conducted using R (R Core Team, Vienna, Austria).[21] Descriptive statistics and t-tests were used to compare baseline characteristics and measures of research productivity between group A and group B hospitals. Univariate logistic regression was utilized to estimate the odds ratio (OR) associated with group A for all measures of research productivity. Multivariate logistic regression was performed, adjusting for significant variables in univariate analysis: mean faculty h-index and number of total (completed and ongoing) clinical trials. Total faculty h-index was excluded from multivariate regression as it was highly correlated with mean faculty publicatication in sensitivity analysis showed a high correlation with mean faculty publications. Linear regression was used to evaluate the association between U.S. News reputation score and measures of research productivity for group A hospitals. Significance levels were set at α = 0.05.

Results

Ophthalmology hospitals were stratified based on their designation as group A or group B and compared with respect to the measures of research productivity ([Table 1]). Relative to group B hospitals, group A hospitals demonstrated significantly higher numbers of total faculty publications, mean faculty publications, and total clinical trials.

Table 1 Characteristics for ophthalmology hospitals listed by the 2020 U.S. News & World Report

Source	Baseline characteristics	Percent group A[a] (n = 12)	Percent group B[b] (n = 26)	Test statistic, p-value
Thiessen et al[17]	Number of faculty Mean (SD)  < 40  ≥ 40	44.42 (16.19) 41.67 (5) 58.33 (7)	35.73 (17.11) 65.38 (17) 34.62 (9)	t = −1.3745, p = 0.0889
Thiessen et al[17]	Total faculty publications (h-index) Mean (SD)  < 400  ≥ 400	672.75 (235.64) 16.67 (2) 83.33 (10)	370.08 (154.10) 53.85 (14) 46.15 (12)	t = −2.2420, p = 0.0156
Thiessen et al[17]	Mean faculty publications (h-index) Mean (SD)  < 15  ≥ 15	15.58 (3.27) 33.33 (4) 66.67 (8)	10.95 (5.20) 76.92 (20) 23.08 (6)	t = −2.7771, p = 0.0043
ClinicalTrials.gov	Total clinical trials Mean (SD)  < 5  ≥ 5	8.42 (5.58) 16.67 (2) 83.33 (10)	3.65 (3.70) 65.38 (17) 34.62 (9)	t = -3.0480, p = 0.0021
NIH RePORTER, BRIMR	Total NIH funding ($) Mean (SD)  < 20,000,000  ≥ 20,000,000	27,078,158.60 (7,762,563.79) 50.00 (5) 50.00 (5)	10,186,560.37 (6,789,173.95) 58.33 (14) 41.67 (10)	t = −0.4339, p = 0.3337
NIH RePORT	Total NEI research funding ($) Mean (SD)  < 20,000,000  ≥ 20,000,000	25,760,449.87 (10,024,040.90) 50.00 (6) 50.00 (6)	12,818,875.48 (8,935,197.86) 57.69 (15) 42.31 (11)	t = −0.4326, p = 0.3339

Abbreviations: BRIMR, Blue Ridge Institute for Medical Research; h-index, Hirsch index; NIH, National Institutes of Health; RePORT, Research Portfolio Online Reporting Tools; RePORTER, RePORT Expenditures and Results; SD, standard deviation; U.S. News, U.S. News & World Report.

^a Group A consists of the hospitals ranked in the top 12 for ophthalmology by U.S. News.

^b Group B consists of the remaining 26 hospitals for ophthalmology listed by U.S. News.

In univariate logistic regression ([Table 2]), variables associated with being in group A included total faculty publications at or above an h-index of 400 (OR: 5.83, 95% confidence interval [CI]: [1.06–32.02]), mean faculty h-index greater than or equal to 15 (OR: 6.67, 95% CI: [1.48–30.11]), and conducting five or more clinical trials (OR: 9.44, 95% CI: [1.69–52.73]). Multivariate logistic regression suggested that mean faculty h-index at or above 15 (OR: 6.13, 95% CI: [1.14–32.94]) and conducting five or more clinical trials (OR: 8.77, 95% CI: [1.39–55.16]) were independently associated with being in group A ([Table 2]). Of note, the total faculty publications variable was excluded in multivariate regression, as it was highly collinear with mean faculty publications.

Table 2 Characteristics of ophthalmology hospitals associated with a top twelve U.S. News reputation-based ranking

Baseline characteristics	Univariate odds ratio (95% CI)	Multivariate adjusted odds ratio (95% CI)
Number of faculty Mean (SD)  < 40  ≥ 40	1 [reference] 2.64 (0.65–10.76)
Total faculty publications (h-index) Mean (SD)  < 400  ≥ 400	1 [reference] 5.83 (1.06–32.02)
Mean faculty publications (h-index) Mean (SD)  < 15  ≥ 15	1 [reference] 6.67 (1.48–30.11)	1 [reference] 6.13 (1.14–32.94)
Total clinical trials Mean (SD)  < 5  ≥ 5	1 [reference] 9.44 (1.69–52.73)	1 [reference] 8.77 (1.39–55.16)
Total NIH funding ($) Mean (SD)  < 20,000,000  ≥ 20,000,000	1 [reference] 1.4 (0.32–6.16)
Total NEI research funding ($) Mean (SD)  < 20,000,000  ≥ 20,000,000	1 [reference] 1.36 (0.35–5.38)

Abbreviations: CI, confidence interval; NEI, National Institutes of Health; NIH, National Eye Institute; SD, standard deviation; U.S. News, U.S. News & World Report.

In linear regression, there was no significant association between U.S. News reputation score and any measures of research productivity (total faculty h-index, mean faculty h-index, total number of clinical trials, NIH funding, and NEI funding) among group A hospitals.

Discussion

Our study found that two objective measures of research productivity were significantly associated with a top-12 U.S. News ranking for the hospital: mean faculty h-index at or above 15 and five or more clinical trials. However, within the top-12 ranked hospitals, measures of research productivity were not associated with U.S. News reputation score. This study suggests the U.S. News ophthalmology ranking—despite being solely based on an opinion poll—may be a good proxy for an ophthalmology department's contribution to research. Our findings may inform the decisions of patients who are selecting a clinic to visit, postgraduate applicants who are interested in research-intensive programs, and physicians who are hoping to become faculty members at a research-intensive institution.

Faculty publications and clinical trials may have a larger influence on hospital reputation than grant funding because they are more publicly visible manifestations of research productivity. Physicians read and subscribe to academic journals[22] [23] and, hence, they may notice when authors from certain institutions have published in high-impact journals. Additionally, many NIH grant recipients were affiliated with large, public institutions that may have had more research-based faculty in basic science. However, practicing ophthalmologists may not follow basic science research as closely as clinical research, reducing its impact on reputation.

This study adds to a mixed body of evidence on U.S. News reputation score. A previous investigation in oncology found a strong correlation between all metrics of research productivity and U.S. News reputation score, while another study in urology identified little correlation between U.S. News' ranking and the authors' research output-based ranking.[9] [10] These disparate results may be due to different statistical methods and measures of research productivity used in each study. The oncology study used simple linear regression analysis and did not exclude collinear variables from the final multiple linear regression,[10] and the urology study did not perform a regression analysis. Both studies relied upon raw faculty publication figures[9] [10] and/or journal impact factors[10]; neither study included an analysis of each publication's impact and citation count.[9] [10]

Limitations

This study has several limitations. We focused on the relationship between reputation and research productivity and did not include other key measures of hospital quality; U.S. News does not publish hospital outcome metrics for ophthalmology.[9] [10] However, many academic institutions consider research a fundamental part of their mission and core values.[24] [25] [26] In addition, using h-index as a measure of academic productivity gives an advantage to older ophthalmologists and more established departments.[27] [28] Moreover, we used nonranked status as a baseline rather than unlisted status; however, this is a more conservative estimate of the differences between ranked and nonranked hospitals. Other factors, such as the number of residency program alumni due to size, longevity of a hospital, or non-NIH financial support for research, may also impact the ranking of a hospital.

Conclusion

In conclusion, this study suggests that research productivity measured by mean faculty h-index of 15 and five or more clinical trials is significantly associated with the reputation-driven U.S. News ophthalmology hospital rankings.

Conflict of Interest

None declared.

Disclaimer

The views expressed here are those of the authors and do not necessarily reflect the position or policy of the U.S. Department of Veterans Affairs or the U.S. government.

• References

• 1 Olmsted MG, Powell R, Murphy J, Bell D, Morley M, Stanley MUS. Methodology: U.S. News & World Report 2019–20 Best Hospitals: Specialty Rankings. Accessed June 24, 2020 at: https://health.usnews.com/media/best_hospitals/Best_Hospitals_Methodology_2019-20
• 2 U.S. News & World Report. FAQ: How and Why We Rank and Rate Hospitals. Accessed June 5, 2020 at: https://health.usnews.com/health-care/best-hospitals/articles/faq-how-and-why-we-rank-and-rate-hospitals
• 3 McGaghie WC. America's best medical schools: a renewed critique of the u.s. news & world report rankings. Acad Med 2019; 94 (09) 1264-1266
• 4 McGaghie WC, Thompson JA. America's best medical schools: a critique of the U.S. News & World Report rankings. Acad Med 2001; 76 (10) 985-992
• 5 Scherer WJ, Danoff S, Schuman JS. Criteria for evaluating ophthalmology departments based on the U.S. News & World Report ranking system. J Clin Acad Ophthalmol. 2016; 8 (01) e1-e9
• 6 Ingram DG, Bachrach BEUS. U.S. News and World Report's rankings of the top 50 children's hospitals for diabetes and endocrinology reflect reputation more than objective measures. J Pediatr Endocrinol Metab 2011; 24 (9-10): 759-761
• 7 Sehgal AR. The role of reputation in U.S. News & World Report's rankings of the top 50 American hospitals. Ann Intern Med 2010; 152 (08) 521-525
• 8 Bush RA, Quigley EJ, Fox L, Garcia-Bassets I. Role of reputation in top pediatric specialties rankings. Pediatrics 2011; 128 (06) 1168-1172
• 9 Kutikov A, Rozenfeld B, Egleston BL, Sirohi M, Hwang RW, Uzzo RG. Academic ranking score: a publication-based reproducible metric of thought leadership in urology. Eur Urol 2012; 61 (03) 435-439
• 10 Prasad V, Goldstein JA. US News and World Report cancer hospital rankings: do they reflect measures of research productivity?. PLoS One 2014; 9 (09) e107803
• 11 Ayanian JZ, Weissman JS, Chasan-Taber S, Epstein AM. Quality of care for two common illnesses in teaching and nonteaching hospitals. Health Aff (Millwood) 1998; 17 (06) 194-205
• 12 Selby P, Gillis C, Haward R. Benefits from specialised cancer care. Lancet 1996; 348 (9023): 313-318
• 13 Yaghoubian A, de Virgilio C, Lee SL. Appendicitis outcomes are better at resident teaching institutions: a multi-institutional analysis. Am J Surg 2010; 200 (06) 810-813 , discussion 813
• 14 Trinh Q-D, Schmitges J, Sun M. et al. Radical prostatectomy at academic versus nonacademic institutions: a population based analysis. J Urol 2011; 186 (05) 1849-1854
• 15 Jacob BA, Lefgren L. The impact of research grant funding on scientific productivity. J Public Econ 2011; 95 (9,10): 1168-1177
• 16 Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005; 102 (46) 16569-16572
• 17 Thiessen CR, Venable GT, Ridenhour NC, Kerr NC. Publication productivity for academic ophthalmologists and academic ophthalmology departments in the United States: an analytical report. J Clin Acad Ophthalmol. 2016; 8: e19-e29
• 18 Birks Y, Fairhurst C, Bloor K, Campbell M, Baird W, Torgerson D. Use of the h-index to measure the quality of the output of health services researchers. J Health Serv Res Policy 2014; 19 (02) 102-109
• 19 Roskoski Jr. R. Ranking Tables of NIH Funding to U.S. Medical Schools. Accessed June 26, 2020 at: http://www.brimr.org/NIH_Awards/NIH_Awards.htm
• 20 Svider PF, Lopez SA, Husain Q, Bhagat N, Eloy JA, Langer PD. The association between scholarly impact and National Institutes of Health funding in ophthalmology. Ophthalmology 2014; 121 (01) 423-428
• 21 R Core Team. The R project for statistical computing. Accessed June 24, 2020 at: https://www.R-project.org/
• 22 Saint S, Christakis DA, Saha S. et al. Journal reading habits of internists. J Gen Intern Med 2000; 15 (12) 881-884
• 23 Tenopir C, King DW, Clarke MT, Na K, Zhou X. Journal reading patterns and preferences of pediatricians. J Med Libr Assoc 2007; 95 (01) 56-63
• 24 Holmes EW, Burks TF, Dzau V. et al. Measuring contributions to the research mission of medical schools. Acad Med 2000; 75 (03) 303-313
• 25 Tarquinio GT, Dittus RS, Byrne DW, Kaiser A, Neilson EG. Effects of performance-based compensation and faculty track on the clinical activity, research portfolio, and teaching mission of a large academic department of medicine. Acad Med 2003; 78 (07) 690-701
• 26 Ramsey PG, Miller ED. A single mission for academic medicine: improving health. JAMA 2009; 301 (14) 1475-1476
• 27 Kreiner G. The slavery of the h-index-measuring the unmeasurable. Front Hum Neurosci 2016; 10: 556
• 28 Costas R, Bordons M. The h-index: advantages, limitations and its relation with other bibliometric indicators at the micro level. J Informetrics 2007; 1 (03) 193-203

Publication History

Received: 09 September 2020

Accepted: 15 December 2020

Article published online:
30 June 2021