Appl Clin Inform 2021; 12(01): 153-163
DOI: 10.1055/s-0041-1722917
Research Article

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned

Zoe Co
1   Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States
A. Jay Holmgren
2   Harvard Business School, Boston, Massachusetts, United States
David C. Classen
3   Department of Clinical Epidemiology, University of Utah, Salt Lake City, Utah, United States
Lisa P. Newmark
4   Clinical and Quality Analysis, Mass General Brigham, Somerville, Massachusetts, United States
Diane L. Seger
4   Clinical and Quality Analysis, Mass General Brigham, Somerville, Massachusetts, United States
Jessica M. Cole
5   Department of Internal Medicine, Division of Epidemiology, University of Utah, Salt Lake City, Utah, United States
Barbara Pon
6   Collaborative Healthcare Patient Safety Organization, Sacramento, California, United States
Karen P. Zimmer
7   Department of Pediatrics, Thomas Jefferson University, Philadelphia, Pennsylvania, United States
David W. Bates
1   Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States
4   Clinical and Quality Analysis, Mass General Brigham, Somerville, Massachusetts, United States
8   Harvard Medical School, Boston, Massachusetts, United States
› Author Affiliations
Funding This study was funded by the Gordon and Betty Moore Foundation.


Background Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed.

Objective To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot.

Methods The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists.

Results For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug–drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot.

Conclusion Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.

Protection of Human and Animal Subjects

No real patients were used in Ambulatory EHR Evaluation Tool, only test patients were used.

Supplementary Material

Publication History

Received: 11 August 2020

Accepted: 16 December 2020

Article published online:
03 March 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385
  • 2 Kuperman GJ, Bobb A, Payne TH. et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (01) 29-40
  • 3 Bates DW, Teich JM, Lee J. et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
  • 4 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
  • 5 Radley DC, Wasserman MR, Olsho LE, Shoemaker SJ, Spranca MD, Bradshaw B. Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems. J Am Med Inform Assoc 2013; 20 (03) 470-476
  • 6 Bates DW, Leape LL, Cullen DJ. et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15) 1311-1316
  • 7 Holmgren AJ, Co Z, Newmark L, Danforth M, Classen D, Bates D. Assessing the safety of electronic health records: a national longitudinal study of medication-related decision support. BMJ Qual Saf 2020; 29 (01) 52-59
  • 8 Classen DC, Holmgren AJ, Co Z. et al. National trends in the safety performance of electronic health record systems from 2009 to 2018. JAMA Netw Open 2020; 3 (05) e205547
  • 9 Denham CR, Classen DC, Swenson SJ, Henderson MJ, Zeltner T, Bates DW. Safe use of electronic health records and health information technology systems: trust but verify. J Patient Saf 2013; 9 (04) 177-189
  • 10 Kaushal R, Kern LM, Barrón Y, Quaresimo J, Abramson EL. Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med 2010; 25 (06) 530-536
  • 11 Gandhi TK, Weingart SN, Seger AC. et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005; 20 (09) 837-841
  • 12 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in EHR-based settings. JAMA Intern Med 2013; 173 (08) 702-704
  • 13 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
  • 14 Co Z, Holmgren AJ, Classen DC. et al. The tradeoffs between safety and alert fatigue: data from a national evaluation of hospital medication-related clinical decision support. J Am Med Inform Assoc 2020; 27 (08) 1252-1258
  • 15 Kilbridge PM, Welebob EM, Classen DC. Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems. Qual Saf Health Care 2006; 15 (02) 81-84
  • 16 Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010; 29 (04) 655-663
  • 17 Leung AA, Keohane C, Lipsitz S. et al. Relationship between medication event rates and the Leapfrog computerized physician order entry evaluation tool. J Am Med Inform Assoc 2013; 20 (e1): e85-e90
  • 18 National Quality Forum. Safe Practices for Better Healthcare-2010 Update: A Consensus Report. 2010
  • 19 The Leapfrog Group. Leapfrog Hospital Survey: Questions & Reporting Periods Endnotes Measure Specifications FAQS. 2020
  • 20 Gandhi TK, Weingart SN, Borus J. et al. Adverse drug events in ambulatory care. N Engl J Med 2003; 348 (16) 1556-1564
  • 21 Phansalkar S, van der Sijs H, Tucker AD. et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 (03) 489-493
  • 22 Office of the National Coordinator for Health Information Technology. Non-federal Acute Care Hospital Electronic Health Record Adoption. Published 2017. Accessed January 11, 2019 at:
  • 23 Ratwani R, Fairbanks T, Savage E. et al. Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation. Appl Clin Inform 2016; 7 (04) 1069-1087
  • 24 Dhillon-Chattha P, McCorkle R, Borycki E. An evidence-based tool for safe configuration of electronic health records: the eSafety checklist. Appl Clin Inform 2018; 9 (04) 817-830
  • 25 Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007; 8 (01) 32
  • 26 Hysong SJ, Sawhney MK, Wilson L. et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak 2011; 11: 22 . Doi: 10.1186/1472-6947-11-22
  • 27 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
  • 28 Scott K, Hathaway E, Sharp K, Smailes P. The development and evaluation of an electronic health record efficiency workshop for providers. Appl Clin Inform 2020; 11 (02) 336-341
  • 29 Wakefield M. Patient safety and quality: an evidence-based handbook for nurses. (The quality chasm series: implications for nursing; ). In: Agency for Healthcare Research and Quality A. ed. Agency for Healthcare Research and Quality, AHRQ. Agency for Healthcare Research and Quality (US); 2008: 1-1403 . Accessed October 28, 2020 at:
  • 30 Wagner MM, Hogan WR. The accuracy of medication data in an outpatient electronic medical record. J Am Med Inform Assoc 1996; 3 (03) 234-244
  • 31 Tamblyn R, Abrahamowicz M, Buckeridge DL. et al. Effect of an electronic medication reconciliation intervention on adverse drug events: a cluster randomized trial. JAMA Netw Open 2019; 2 (09) e1910756
  • 32 Mekonnen AB, Abebe TB, McLachlan AJ, Brien JAE. Impact of electronic medication reconciliation interventions on medication discrepancies at hospital transitions: a systematic review and meta-analysis. BMC Med Inform Decis Mak 2016; 16 (01) 112