A New EHR Training Curriculum and Assessment for Pediatric Residents

Abstract

Background Conventional classroom Electronic Health Record (EHR) training is often insufficient for new EHR users. Studies suggest that enhanced training with a hands-on approach and closely supported clinical use is beneficial.

Objectives Our goals were to develop an enhanced EHR learning curriculum for Post Graduate Year 1 (PGY1) residents and measure changes in EHR skill proficiency, efficiency, and self-efficacy.

Methods A novel three-phase, multimodal enhanced EHR curriculum was designed for a cohort of PGY1 residents. After basic training, residents began phase 1 of enhanced training, including demonstrations, live practice, and order set review. Phase 2 involved skills-oriented assignments, role playing, and medication entry. Phase 3 included shadowing, scribing histories, and supervised order entry. Residents' EHR skills and attitudes were measured and compared before and after the enhanced curriculum via proficiency test and a survey of efficiency and self-efficacy.

Results Nineteen of 26 PGY1 residents participated in the study (73%). There was significant improvement in mean proficiency scores and two of the five individual proficiency scores. There were significant improvements in most efficiency survey responses from pre- to postintervention. For the self-efficacy presurvey, many PGY1s reported to be “very” or “somewhat confident” performing each of the five tasks, and perceptions did not improve or worsened on most postsurvey responses. The greatest resource was the time required to design and deliver the enhanced training.

Conclusion An enhanced training curriculum along with a proficiency assessment was developed and described here. An enhanced training curriculum significantly improved PGY1 EHR efficiency and some measures of proficiency but not self-efficacy. This intervention may support improved EHR-related clinic workflows, which ultimately could enable residents and preceptors to prioritize patient care and time for clinical education.

Keywords

electronic health record - EHR - research - resident - EHR training - primary care - learner - assessment

Background and Significance

The Electronic Health Record (EHR) has improved many aspects of patient care, including quality, efficiency, and cost but has significantly impacted provider–patient dynamics and clinical workflow.[1] [2] In a recent national survey, Graduate Medical Educators agreed that the EHR impacts the education of residents and fellows in several ways, especially in the importance of “balancing focus on electronic documentation with patient engagement,” and the need to teach EHR use through hands-on practice.[3] Many EHR training programs are technical, brief, and not customized to the learner's clinical specialty. Trainer experience and learner engagement vary. In our experience, residents do much of their EHR learning during clinical care time, adopting the habits and workarounds of their colleagues. As resident physicians are central to academic institutions, suboptimal EHR training can result in inefficiency and user frustration and the need for significant ongoing support.[4] [5] [6] Such delays interrupt patient care, increase workload, and residents have reported decreased available education time.[4] This is worrisome, as command of information technology is “vital in the modern practice of medicine.”[7]

Effective EHR learning programs incorporate social cognitive theory and adult learning principles.[8] [9] Several studies suggest that enhanced EHR training positively impacts EHR adoption, efficiency, and patient care.[5] [10] Enhanced EHR training has been investigated for large groups of residents across multiple specialties[4] [11] [12] and medical students,[13] [14] but objective measurement of these interventions is absent or in early phases of exploration.[11] [15] [16] Most published literature on this topic only utilizes self-assessment methods.

Objectives

• To design and implement an enhanced EHR learning curriculum that incorporates adult learning theory and multimodal educational tools relevant to the PGY1 resident.

• To create and implement assessment tools for EHR proficiency and self-reported efficiency and self-efficacy.

Methods

This study was completed at a large academic pediatric practice. All new PGY1 residents received basic training taught by a nonclinical software specialist. Basic training occurred in a classroom setting with individual desktop computers and a large instructional monitor. Content was not specialty specific and was presented in a didactic fashion with occasional individual learner task practice. In the months following initial go-live in January 2012, EHR superusers conducted an informal needs assessment in the resident clinic. Superusers were attending physicians who had received additional EHR training, led site implementation and clinical support, and were regarded as local experts. Superusers noted that in the months after EHR adoption, preceptors were devoting approximately 30 to 60 minutes per clinic session to instructing residents in EHR use and correcting EHR errors, instead of providing clinical education. Superusers also noted widespread deficiencies in residents' knowledge of basic EHR tasks and decreased time spent on clinical care. In response to the needs assessment, superusers created an enhanced EHR training curriculum for incoming residents built upon social cognitive theory and adult learning principles frequently used in graduate medical education (see Appendix 1).

Our study was a prospective, single center cohort study of Pediatric PGY1s' EHR skills and attitudes before and after the enhanced EHR training curriculum. All new PGY1s were required to participate in the basic and enhanced training, and all were invited to participate in this study from the start. Written consent was obtained for those who agreed. The EHR utilized in our practice was NextGen Ambulatory EHR (QSI Management, LLC; Horsham, Pennsylvania, United States).

Enhanced Training Intervention

We designed an enhanced, three-phase EHR training curriculum to follow basic training (Appendix 2).

Phase 1 was designed to orient PGY1s to EHR layout and basic visit workflows. An EHR physician superuser led small group classes over 4 hours, reviewing a typical pediatric visit using demonstrations, learner practice activities and role plays, and entry of medications and vaccines. For example, a volunteer pretended to be a parent, using a script to give the history for a sick child, while PGY1 learners documented within the Histories template. In another example, PGY1s were provided a list of 10 frequently prescribed pediatric medications (such as amoxicillin suspension) and practiced creating prescriptions within the medication module. All EHR terminology was defined in language familiar to learners.

Phase 2 was held within the PGY1's continuity clinic to situate an EHR practice experience within the actual clinical setting. Residents were provided 60 minutes during the first week's orientation to strengthen and reinforce EHR skills through role playing with data entry. For example, a pair of PGY1s would use an exam room to role play as patient and physician, with the physician entering data into the Histories, Review of Systems, Physical Exam, and Assessments templates. PGY1s also utilized this time to practice creating prescriptions that were then reviewed by an attending physician, and later set up saved “favorite” medications, My-Phrases, and templates within the system.

Phase 3 was held during the PGY1s' second clinic week, focusing on skill application with supervision and feedback. As real-time, one-on-one EHR training in the work environment has been shown to enhance performance,[6] PGY1s observed two patient care visits performed by their senior residents (PGY3) and then acted as scribes for subsequent visits. As PGY3 residents had been using the system for 2 years, they served as experienced peer educators to support new learners' skill acquisition during this session.[8] PGY1 residents also entered orders and completed electronic forms under supervision during the PGY3 precepting time.

Assessment of Residents

EHR skills were measured through a proficiency assessment tool, and surveys measured self-reported EHR efficiency and self-efficacy. The proficiency assessment and surveys were administered together on two separate occasions: following basic training (pretest) and following enhanced training (posttest, [Fig. 1]).

Fig. 1 Schematic of curriculum and evaluation.

Proficiency Assessment Tool

To our knowledge, there exists no validated, widely used measure of EHR proficiency. To create our own measure and establish that its content represented the appropriate skills,[17] six typical EHR tasks for pediatric primary care were assigned to experienced clinicians for evaluation ( Appendix 3) and adjusted so that they could be completed within 6 minutes. Each task was given a score of 0 (not done), 1 (partially done), or 2 (completely done) and equally weighted toward a total score from 0 to 12. Each task was preassigned a detailed definition of the three possible scores, including attention to quality of content. At the pre- and posttests, each PGY1 was provided 6 minutes to complete the six tasks. Results were scored by a clinician blinded to curriculum content and study objectives. Mean group scores were utilized for analysis.

Efficiency and Self-Efficacy Surveys

Since no single, validated, widely used tool existed to measure these data, question themes were culled from the literature. Questions were piloted with physicians in the practice (details in Appendix 3). Five efficiency questions used the stem, “I can correctly complete [EHR task] without asking someone for help...” with a 4-option Likert response scale. Five self-efficacy questions used the stem, “How confident do you feel doing the following tasks...?” with a 4-option Likert response scale.

Analysis

For pre- and postcurriculum results, paired t-tests were employed to compare mean proficiency assessment scores, and the Wilcoxon Signed Rank Test (WSRT) compared the survey differences. Only assessments with matching pre- and postresponses were included. Survey data were imported into SPSS for analysis. Assuming 25 participants, we estimated 80% power to detect a change of ≥ 1.5 points on each scale with an estimated standard deviation of 2.5 and an α of 0.05 using a 2-sided one sample t-test.

Results

The incoming class of 26 PGY1s initiated the basic and enhanced training and consented to study participation. One PGY1 could not complete the training due to illness, and six individuals did not complete both pre- and posttests. This resulted in a sample size of 19 of 26 or 73%.

Proficiency Assessment

The mean total proficiency scores improved with statistical significance from pre- to posttesting (6.8–7.8, p = 0.048; [Fig. 2]). Of the six items, there was statistically significant improvement in medication entry (mean: 1.0–1.7, p = 0.02) and completing school forms (mean: 0.2–0.7, p = 0.03) with nonsignificant changes in the remaining items (adding vital signs, adding allergies, adding diagnoses, and ordering vaccinations).

Fig. 2 Mean proficiency assessment scores pre- and postcurriculum for six tasks and total for all tasks. *p < 0.05.

Efficiency and Self-Efficacy

Residents reported a low sense of efficiency and self-efficacy on the pretest. For pretest efficiency, only a minority of residents reported that they could correctly complete tasks “always” or “most of the time” ([Fig. 3]). For pretest self-efficacy, a minority of residents felt “very confident” or “somewhat confident” in three of the tasks, although they reported confidence while adding allergies or documenting a telephone call ([Fig. 4]). In the posttest, most residents reported a higher sense of efficiency than on the pretest, including vaccine ordering (WSRT, p < 0.05), complete blood count (CBC ordering; WSRT, p < 0.05), school physical form completion (WSRT, p < 0.05), and updating patient location (WSRT, p < 0.05; [Fig. 3]). History taking did not improve significantly (WSRT, p = 0.074). Residents reported more self-efficacy on the posttest for adding saved phrases (WSRT, p < 0.05), but the remainder reported less or stayed the same, including documenting a telephone call (WSRT, p < 0.05), adding allergies (WSRT, p = 1.00), ordering a rapid strep test (WSRT, p = 0.49), and locating test results (WSRT, p = 0.22; [Fig. 4]).

Fig. 3 Efficiency survey results pre- and post-curriculum for five tasks. Question stems, “I can correctly complete (task) without asking someone for help…” *p < 0.05.

Fig. 4 Self-efficacy survey results pre- and postcurriculum for five tasks. Question stems, “How confident do you feel doing the following tasks?” *p < 0.05.

There were similarities between results for those activities that were measured by both the surveys and the proficiency test. For updating allergies, most respondents reported confidence and had high proficiency scores at both the pre- and postassessments ([Figs. 2] and [4]). For ordering vaccines, respondents reported very low efficiency at the pretest and made modest improvements on the posttest ([Fig. 3]). In a similar pattern, proficiency scores were low and did not improve ([Fig. 2]). For school form completion, residents first expressed low efficiency and had low proficiency scores, and both measures significantly improved from pre- to postassessments ([Figs. 2] and [3]).

Discussion

As suspected, participating PGY1 residents did not report substantial efficiency or self-efficacy after basic EHR training. To develop an enhanced curriculum for the use after basic training, we incorporated additional adult learning theory, simulation, physician trainers, and one-on-one experiences.

These results suggest that a three-phased enhanced training curriculum may have improved resident EHR efficiency and some measures of proficiency. These findings are relevant given that EHR competence is now a basic clinical skill, and residents who lack proficiency and efficiency may struggle to perform clinical tasks. Although medicine and medical education are highly regulated fields, the literature about measuring EHR skill acquisition is scarce. This article adds to the field by describing a novel way to assess EHR skill acquisition. This study could contribute to future development of standardized training and objective assessment of EHR provider–users across multiple primary care settings. Of note, the biggest resource necessary for our intervention was physician time (4 × 2 = 8 hours), as EHR superuser attending physicians delivered the curriculum. This time resource, however, was less than the estimated amount of time required for on-the-spot EHR assistance provided by attending physicians for residents prior to this study.

Certain outcome measures showed no improvement or worsened. Although two individual proficiency scores and the average proficiency scores showed improvement, several scores worsened, which may reflect differing complexity of particular tasks. Additionally, the three of the four tasks that showed no improvement in proficiency were relatively simple tasks (vital signs, allergies, and diagnosis) that had high initial scores. It may be that learners achieved mastery after basic training and therefore had little room to improve further. Worsening self-efficacy may be due to the complex user interface of the EHR application. Also, as large parts of phases 2 and 3 were peer-to-peer, the quality of the training interaction was not established. In addition, it is possible that experienced users in phase 3 passed on maladaptive workarounds, worsening learner scores. Lastly, residents also may reach a “saturation point” of new information during a short but demanding orientation period.

Related studies published on this topic have mixed results. Dastagir and colleagues surveyed clinicians already experienced in their EHR who received a proficiency training course. Comparing pre- and posttraining surveys, they demonstrated improvements in self-assessed efficiency.[5] Vuk and colleagues demonstrated improved self-efficacy assessed before and after simulation training,[10] which differed from our self-efficacy results. Jalota and colleagues conducted a real-time peer training intervention with physicians already experienced in their EHR and showed improved documentation efficiency over the control group.[6] Reis and colleagues demonstrated no improvement in attitude, sense of competence, or observed proficiency when comparing a simulation based training to traditional lecture format.[12]

Our study had several limitations. Self-report of efficiency might not have reflected actual efficiency. Our sample size was small, and performance at a single center might have limited its external validity. A lack of a control group limited our ability to attribute changes to the intervention instead of increased training exposure alone (maturity threat). A selection bias might have been introduced by including only new residents, who typically were younger and more comfortable with technology than other physicians. The majority of our assessments dealt with data entry tasks, even though data extraction and interpretation were important EHR skills. Although our curriculum was adapted to our EHR application, it could be easily adapted to other systems.

Single group pre-and posttest models, such as ours, are at a risk of testing threat, in which exposure to the pretest influences participants' learning, thereby exaggerating their performance on the posttest. Our proficiency assessment and surveys evaluated slightly different sets of EHR tasks, therefore evaluating the exact set of tasks through the lens of both the surveys and the proficiency tool might have allowed the curriculum to be refined and focused on tasks requiring more instruction to achieve mastery. The assessment tools also deserve more thorough validation, including against clinical outcomes. However, this study was an attempt to integrate adult learning principles into EHR training and describe a method of EHR skill assessment in a manner that could be replicated. Study leaders received verbal feedback from colleagues describing stronger PGY1 EHR skills and the additional time available for precepting and clinical education after this intervention. Future studies could formally measure this preceptor feedback and measure time available for EHR teaching.

Next steps in this project would involve pretraining baseline assessments and further development of the proficiency measure. Recruiting a control group at a comparable institution, repeating the posttest at a later date, and monitoring improvements in accuracy and efficiency during subsequent clinical care would enrich our understanding of the effects of the intervention.

Conclusion

This study details an EHR training intervention that may improve residents' EHR skills; it proposes several measures of EHR skill proficiency and efficiency that may be studied further. Ultimately, an enhanced EHR training curriculum should strengthen residents' EHR skills and decrease interruptions, allowing residents and preceptors to prioritize direct patient care and build clinical knowledge.

Clinical Relevance Statement

Residents and EHRs build the backbone of academic medical institutions; however, resident EHR training is often overlooked. This study presents a novel, enhanced EHR training intervention designed with adult learning principles, and measures learners' proficiency, efficiency, and self-efficacy. Significant improvements in efficiency and some measure of proficiency are noted, which could support streamlined workflows, patient care quality, and time for clinical education.

Multiple Choice Questions

1. Training curricula for clinicians to use an EHR should include

   • A fast-paced style to keep the learner's attention

   • Careful attention to the concerns about the application raised by the learners

   • Well-trained mock patients

   • Active learning techniques using real-life scenarios

2. When compared with standard EHR training, enhanced EHR training

   • Can positively impact the efficiency of clinicians' EHR use

   • Incorporates adult learning principles and/or social cognitive theory

   • Can require more time to complete

   • All of the above

Answers

1. Correct answer is D, Active learning techniques using real-life scenarios. Adult learning theory uses real-life scenarios and active engagement of the learner. The quality of the mock patient, being fast paced to keep the learner's attention and time spent on learners' concerns of the EHR application design are less important components of EHR training for medical providers.

2. Correct answer is D, All of the above. Incorporation of adult learning principles can result in a longer training period when compared with out-of-the-box training. However, evidence suggests that this could result in increased EHR user efficiency.

Conflict of Interest

None.

Acknowledgments

The authors gratefully acknowledge Sarah Long, MD, Hans Kersten, MD, and Nancy Spector, MD, for their guidance and key editing of the manuscript.

Protection of Human and Animal Subjects

This study was reviewed and approved by the Institutional Review Board of the Drexel University College of Medicine and was in compliance with the World Medical Association Declaration of Helsinki on ethical principles regarding medical research involving human subjects.

• References

• 1 Tierney MJ, Pageler NM, Kahana M, Pantaleoni JL, Longhurst CA. Medical education in the electronic medical record (EMR) era: benefits, challenges, and future directions. Acad Med 2013; 88 (06) 748-752
• 2 Keenan CR, Nguyen HH, Srinivasan M. Electronic medical records and their impact on resident and medical student education. Acad Psychiatry 2006; 30 (06) 522-527
• 3 Atwater AR, Rudd M, Brown A. , et al. Developing teaching strategies in the EHR era: a survey of GME experts. J Grad Med Educ 2016; 8 (04) 581-586
• 4 Nuovo J, Hutchinson D, Balsbaugh T, Keenan C. Establishing electronic health record competency testing for first-year residents. J Grad Med Educ 2013; 5 (04) 658-661
• 5 Dastagir MT, Chin HL, McNamara M, Poteraj K, Battaglini S, Alstot L. Advanced proficiency EHR training: effect on physicians' EHR efficiency, EHR satisfaction and job satisfaction. AMIA Annu Symp Proc 2012; 2012: 136-143
• 6 Jalota L, Aryal MR, Mahmood M, Wasser T, Donato A. Interventions to increase physician efficiency and comfort with an electronic health record system. Methods Inf Med 2015; 54 (01) 103-109
• 7 Benson B. Competency 7. Use information technology to optimize learning and care delivery. Within: the pediatrics milestones: pursuit of a national system of workplace-based assessment through key stakeholder collaboration. Acad Pediatr 2014; 14: S48-S50
• 8 McAlearney AS, Robbins J, Kowalczyk N, Chisolm DJ, Song PH. The role of cognitive and learning theories in supporting successful EHR system implementation training: a qualitative study. Med Care Res Rev 2012; 69 (03) 294-315
• 9 Reed S, Shell R, Kassis K. , et al. Applying adult learning practices in medical education. Curr Probl Pediatr Adolesc Health Care 2014; 44 (06) 170-181
• 10 Vuk J, Anders ME, Mercado CC, Kennedy RL, Casella J, Steelman SC. Impact of simulation training on self-efficacy of outpatient health care providers to use electronic health records. Int J Med Inform 2015; 84 (06) 423-429
• 11 Milano CE, Hardman JA, Plesiu A, Rdesinski RE, Biagioli FE. Simulated electronic health record (Sim-EHR) curriculum: teaching EHR skills and use of the EHR for disease management and prevention. Acad Med 2014; 89 (03) 399-403
• 12 Reis S, Sagi D, Eisenberg O. , et al. The impact of residents' training in Electronic Medical Record (EMR) use on their competence: report of a pragmatic trial. Patient Educ Couns 2013; 93 (03) 515-521
• 13 Wald HS, George P, Reis SP, Taylor JS. Electronic health record training in undergraduate medical education: bridging theory to practice with curricula for empowering patient- and relationship-centered care in the computerized setting. Acad Med 2014; 89 (03) 380-386
• 14 Biagioli FE, Elliot DL, Palmer RT. , et al. The electronic health record objective structured clinical examination: assessing student competency in patient interactions while using the electronic health record. Acad Med 2017; 92 (01) 87-91
• 15 Pantaleoni JL, Stevens LA, Mailes ES, Goad BA, Longhurst CA. Successful physician training program for large scale EMR implementation. Appl Clin Inform 2015; 6 (01) 80-95
• 16 Nicklaus J, Kusser J, Zessin J, Amaya M. Transforming education for electronic health record implementation. J Contin Educ Nurs 2015; 46 (08) 359-363
• 17 Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003; 37 (09) 830-837