Keywords
medical education - residency selection - SF match - match process - USMLE - Step
1 - program directors
Ophthalmology is among the most competitive specialties, as each year applicants outnumber
the positions available, resulting in a match rate between 72 and 78% since 2011.[1] In turn, United States Medical Licensing Examination (UMSLE) Step 1 scores for ophthalmology
applicants are higher on average compared with other specialties and have increased
nearly one standard deviation over the past 10 years.[1]
[2] To be competitive, students apply to more programs each year and this trend has
resulted in an increase in the average number of applications per applicant from 52
to 77 since 2011.[1] This increase in applicants has presented ophthalmology program directors (PDs)
with several challenges in selecting candidates and has led to a greater reliance
on objective measures, such as grades, class rank, and USMLE scores.[3] In a 2017 survey of ophthalmology PDs, the UMSLE Step 1 score was found to be among
the most important factors for screening applicants for interview.[4]
In February 2020, it was announced that USMLE Step 1 score reporting will change from
the traditional three-digit numeric score to reporting only a pass/fail outcome as
soon as January 2022. A national survey of PDs from across many specialties highlighted
the generally negative reactions to the updated scoring system.[5] Yet, the viewpoint of ophthalmology PDs remains uncharacterized. This study aims
to clarify the perception among ophthalmology PDs of binary USMLE Step 1 scoring.
Methods
To characterize how ophthalmology PDs will respond to pass/fail USMLE Step 1 scoring,
a 19-item survey was validated through phases of prepilot and pilot testing. The survey
was iteratively revised, and internal validity was assessed by calculation of the
Cronbach's α (0.87). Institutional Review Board/Ethics Committee ruled that approval was not required
for this study.
Directors of Accreditation Council for Graduate Medical Education (ACGME)-accredited
residency programs in 30 specialties were invited to participate, including 111 ophthalmology
PDs.[5] Email addresses were obtained from publicly available ACGME documents. Over the
course of 4 weeks, individualized requests for participation were sent through the
REDCap (Research Electronic Data Capture) platform.
Descriptive statistics were computed using Microsoft Excel. All values are listed
as discrete numbers or percentages with 95% confidence intervals. Statistical significance
in response plurality was determined by nonoverlapping 95% confidence intervals.
Results
A total of 56 ophthalmology PDs participated in the survey, representing 44.8% of
the ACGME-accredited residency programs. The majority of responses were evenly distributed
among programs from the Northeast (32.1%), South (26.8%), West (14.3%), and Midwest
(26.8%). The median PD age was 48 years and the majority were male (71.4%); the average
tenure as PD was 7.1 years ([Table 1]). This sex distribution and average tenure among the ophthalmology PDs was consistent
with the characteristics of respondents from all specialties.[5]
Table 1
Demographic characteristics of ophthalmology residency program director survey respondents
|
Characteristics
|
Participants, no. (%)
|
|
Age, y, median (IQR)
|
48 (41–58)
|
|
Male gender
|
40 (71.4)
|
|
Tenure in position, y, median (IQR)
|
5 (3–10)
|
|
Region
|
|
|
Northeast
|
18 (32.1)
|
|
South
|
15 (26.8)
|
|
Midwest
|
15 (26.8)
|
|
West
|
8 (14.3)
|
Only 10.7% of PD respondents felt that changing the USMLE Step 1 to pass/fail was
a good idea, and the majority (92.9%) anticipate it will become more difficult to
objectively compare applicants. Furthermore, the elimination of numerical Step 1 reporting
will increase their emphasis on the USMLE Step 2 Clinical Knowledge (CK) scores, and
76.9% of PDs will require applicants to submit Step 2 CK scores at the time of application.
Responses indicate that this shift in reporting will likely lead to increased emphasis
on an applicant's medical school reputation during the application process (70.4%)
and negatively affect international medical graduates (73.2%). When asked if the change
to pass/fail grading would improve student well-being, the majority of PDs were neutral
(46.4%). Lastly, only 9.1% supported a similar change to pass/fail for Step 2 CK scoring
reports ([Table 2]).
Table 2
Ophthalmology program director perspectives of scoring Step 1 pass/fail
|
Statement
|
Disagree
|
Neutral
|
Agree
|
Summary of responses from all specialties[5]
|
|
Percent (95% confidence interval)
|
|
Changing the USMLE Step 1 to pass/fail:
|
|
Is a good idea
|
66.1 (53.7–78.5)[a]
|
23.2 (12.2–34.3)
|
10.7 (02.6–18.8)
|
Disagree – 60.8 (58.8–63.6)[a]
|
|
Will make it more difficult to objectively compare applicants
|
5.4 (0.0–11.3)
|
1.8 (0.0–5.3)
|
92.9 (86.1–99.6)[a]
|
Agree – 77.2 (74.8–79.6)[a]
|
|
Will increase emphasis on Step 2 CK scores in selecting applicants for my program
|
5.4 (0.0–11.3)
|
12.5 (3.8–21.2)
|
82.1 (72.1–92.2)[a]
|
Agree – 80.7 (78.4–82.0)[a]
|
|
Will put international medical graduates at a disadvantage
|
12.5 (3.8–21.2)
|
14.3 (5.1–23.5)
|
73.2 (61.6–84.8)[a]
|
Agree – 44.4 (41.6–47.3)[a]
|
|
Will decrease socioeconomic disparities in the application process
|
42.9 (29.9–55.8)
|
42.9 (29.9–55.8)
|
14.3 (5.1–23.5)
|
Nonsignificant
|
|
Will decrease medical student knowledge of the basic sciences
|
23.2 (12.2–34.3)
|
46.4 (33.4–59.5)
|
30.4 (18.3–42.4)
|
Nonsignificant
|
|
Will improve medical student well-being
|
23.2 (12.2–34.3)
|
46.4 (33.4–59.5)
|
30.4 (18.3–42.4)
|
Nonsignificant
|
|
Will make applicant screening more arduous
|
3.6 (0.0–8.4)
|
1.8 (0.0–5.3)
|
94.6 (88.7–100.0)[a]
|
–
|
|
As a result of changing USMLE Step 1 to pass/fail:
|
|
I will now require applicants to submit Step 2 CK scores with ERAS/CAS
|
7.7 (0.4–14.9)
|
15.4 (5.6–25.2)
|
76.9 (65.5–88.4)[a]
|
Agree – 77.1 (74.7–79.5)[a]
|
|
Where an applicant goes to medical school will be more important in screening and
selection for my program
|
13.0 (4.0–21.9)
|
16.7 (6.7–26.6)
|
70.4 (58.2–82.5)[a]
|
Agree – 56.8 (54.0–59.7)[a]
|
|
Step 2 CK should also be changed to pass/fail
|
80.0 (69.4–90.6)[a]
|
10.9 (2.7–19.1)
|
9.1 (1.5–16.7)
|
Disagree – 60.8 (58.8–63.6)[a]
|
Abbreviations: CAS, Central Application Service; CK, Clinical Knowledge; ERAS, Electronic
Residency Application Service; USMLE, United States Medical Licensing Examination.
a Indicates a statistically significant plurality of responses by nonoverlapping confidence
intervals.
Discussion
In this analysis, ophthalmology PDs generally disapproved of the change to pass/fail
Step 1 scoring. The USMLE Step 1 exam is the first of a three-part exam series originally
established for medical licensure decisions but over time has evolved to include many
secondary functions, including selection of residency applicants and honor society
members.[6] The USMLE scoring reform was intended to improve the transition from undergraduate
medical education to graduate medical education while balancing student learning and
well-being.[7] Specifically, the change was aimed at reducing the current perceived overemphasis
on USMLE performance, which has become particularly prevalent in competitive specialties
such as ophthalmology. However, ophthalmology PDs are skeptical of the predicted benefits
with several concerns raised regarding possible unintended consequences of this isolated
change.
Another impetus for change was to address the widening of socioeconomic and racial
disparities in the application process. Step 1 scores have been found to vary by race,
gender, and socioeconomic status.[8] While removing Step 1 scores may eliminate some of the disparities associated with
the exam, PDs were still overall undecided about the impact on socioeconomic disparities
in the application process. This may be because eliminating numerical Step 1 scores
will simply lead to increased focus on Step 2 CK, a test with similar problems.[9] Yet, most PDs disagreed (80.0%) with implementing a similar scoring change to the
Step 2 CK exam, highlighting a need for objective measures. With most PDs (70.4%)
indicating that binary Step 1 scoring will result in more weight being placed on medical
school reputation, the socioeconomic disparities may remain unchanged and international
medical graduates may be negatively affected. Top medical schools and institutions
with greater National Institutes of Health funding are more likely to have resources
and mentors for students to engage in research, a widely known metric in the screening
and review process that may become more important after the scoring change.[3] Students at these institutions are also more likely to have an ophthalmology department
and access to well-known faculty within the specialty. This increased reliance on
medical school reputation may adversely impact attempts to improve diversity within
the specialty.
While the negative perceptions of a pass/fail Step 1 are not unique to the specialty,[5] the ophthalmology PDs represented much more concordant and decisive opinions when
compared with the general perspective of all specialties ([Table 2]). This is no surprise as the ophthalmology match process is positioned to be more
drastically affected due to the early match timeline. Applicants for ophthalmology
begin registration in early summer and typically submit applications by mid-August.
As a result of this timeline, Step 2 CK scores are usually not available in time for
the screening and selection process. In fact, only 46% of ophthalmology applicants
in 2011 reported their Step 2 score.[10] Because of the increased emphasis on Step 2 CK, medical students may need to complete
this exam before the conclusion of their third year. This shortened timeframe may
place a financial burden on students from lower socioeconomic backgrounds as they
will now be required to pay registration fees (over $1,000 combined) for both exams
within the same year. Students will also feel more pressure to perform well on Step
2 CK knowing that there will no longer be a “second chance” to redeem initial performance.
This academic stress may also disrupt clinical rotations and other curricular programs
as students prepare for Step 2 CK.
Current literature regarding the use of Step 2 CK as a residency selection tool and
predictor of resident performance is scarce—a concerted effort needs to be dedicated
to interpreting and understanding these results in the context of ophthalmology training.
Additional consideration should be focused on possible reforms to the match process
that will improve applicants' ability to present their best attributes while promoting
diversity and inclusion within the specialty. Though progress has been made toward
this goal, no single clear method has emerged.[11] Further research is necessary to aid PDs and admission committees to find creative,
evidence-based methods to evaluate applicants holistically. While the intention of
the Step 1 reform was to promote a more comprehensive review of applicants beyond
standardized test performance, an unintentional consequence may be that more emphasis
is placed on factors that further disadvantage groups already underrepresented in
the field.
Limitations of this study include possible self-selection bias as PDs with more opinionated
views on the topic may be more likely to respond to the online survey. While the response
rate and sample size may limit generalizability, this study still provides the largest
sample size (to the authors' knowledge) of responses from PDs with regard to the new
Step 1 scoring change. Future investigations should assess additional characteristics
of PDs and their respective programs to understand the correlation between survey
responses and factors such as research funding and activity, program size, and ranking
success.
Conclusion
In summary, the majority of ophthalmology PD respondents do not support pass/fail
Step 1 scoring, citing the lack of benefit on student well-being, shifted emphasis
on Step 2 CK, and greater burden of residency selection. An isolated change to Step
1 scores without concurrent reform of the application may be detrimental to those
involved in the ophthalmology early match process.
