Usability Evaluation of An Electronic Medication Administration Record (eMAR) Application

Summary

Background: Electronic medication administration records (eMARs) have been widely used in recent years. However, formal usability evaluations are not yet available for these vendor applications, especially from the perspective of nurses, the largest group of eMAR users.

Objective: To conduct a formal usability evaluation of an implemented eMAR.

Methods: Four evaluators examined a commercial vendor eMAR using heuristic evaluation techniques. The evaluators defined seven tasks typical of eMAR use and independently evaluated the application. Consensus techniques were used to obtain 100% agreement of identified usability problems and severity ratings. Findings were reviewed with 5 clinical staff nurses and the Director of Clinical Informatics who verified findings with a small group of clinical nurses.

Results: Evaluators found 60 usability problems categorized into 233 heuristic violations. Match, Error, and Visibility heuristics were the most frequently violated. Administer Medication and Order and Modify Medications tasks had the highest number of heuristic violations and usability problems rated as major or catastrophic.

Conclusion: The high number of usability problems could impact the effectiveness, efficiency and satisfaction of nurses’ medication administration activities and may include concerns about patient safety. Usability is a joint responsibility between sites and vendors. We offer a call to action for usability evaluations at all sites and eMAR application redesign as necessary to improve the user experience and promote patient safety.

Keywords

Interfaces and usability - Human-computer interaction - Medication administration records - Electronic health record - Testing and evaluation

 

Conflict of Interest

None of the authors have actual or potential conflicts of interest to disclose.

• References

• 1 Agrawal A, Aronson JK, Britten N, Ferner RE, de Smet PA, Fialova D. et al. Medication errors: problems and recommendations from a consensus meeting. Br J Clin Pharmacol 2009; 67: 592-598.
• 2 Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001; 8: 299-308.
• 3 Institute of Medicine.. To err is human; Building a safer health system. Washington, DC: National Academy Press; 2000
• 4 Hurley AC, Bane A, Fotakis S, Duffy ME, Sevigny A, Poon EG. et al. Nurses’ satisfaction with medication administration point-of-care technology. J Nurs Adm 2007; 37: 343-349.
• 5 Cook RI, O’Connor M, Render ML, Woods DD. Operating at the sharp end: The human factors of complex technical work and its implications for patient safety. In: Manuel BM, Nora PF editors. Surgical patient safety: Essential information for surgeons in today’s environment. Chicago, IL: American College of Surgeons; 2004. p. 19-30.
• 6 Wreathall J, Reason J. editors. Human errors and disasters. Fifth IEEE Conference on Human Factors and Power Plants 1992 June 7-11 Monterey, CA.: IEEE.
• 7 Potter P, Wolf L, Boxerman S, Grayson D, Sledge J, Dunagan C. et al. An analysis of nurses’ cognitive work: A new perspective for understanding medical errors. AHRQ Publication No. 05–0021–1. K H, JB B, Marks E, Lewin DI editors. Rockville, MD: Agency for Healthcare Research and Quality; 2005.
• 8 Hicks R, Becker S, Cousins D. MEDMARX Data Report. A report on the relationship of drug names and medication errors in response to the Institute of Medicine’s Call for Action. Rockville, MD: Center for the Advancement of Patient Safety, US Pharmacopeia.; 2008
• 9 Zheng K, Padman R, Johnson MP, Diamond HS. An interface-driven analysis of user interactions with an electronic health records system. J Am Med Inform Assoc 2009; 16: 228-237.
• 10 Armijo D, McDonnell C, Werner K. Electronic health record usability: Evaluation and use Ccase framework. AHRQ Publication No. 09(10)-0091–1-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2009
• 11 Armijo D, McDonnell C, Werner K. Electronic health record usability: Interface design considerations. AHRQ Publication No. 09(10)-0091–2-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2009
• 12 Staggers N, Kobus D, Brown C. Nurses’ evaluations of a novel design for an electronic medication administration record. Comput Inform Nurs 2007; 25: 67-75.
• 13 Huang H, Lee TT. Evaluation of ICU Nurses’ use of the clinical information system in Taiwan. Comput Inform Nurs 2010; 29: 221-229.
• 14 Beuscart-Zéphir MC, Pelayo S, Degoulet P, Anceaux F, Guerlinger S, Meaux JJ. A usability study of CPOE medication administration functions: impact on physician-nurse cooperation. Medinfo 2004; 11: 1018-1022.
• 15 Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 (Suppl. 01) S69-S76.
• 16 Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio J, Ma’luf N. et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med 1999; 106: 144-150.
• 17 Cordero L, Kuehn L, Kumar RR, Mekhjian HS. Impact of computerized physician order entry on clinical practice in a newborn intensive care unit. J Perinatol 2004; 24: 88-93.
• 18 Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Jama 2005; 293: 1223-1238.
• 19 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163: 1409-1416.
• 20 Mekhjian HS, Kumar RR, Kuehn L, Bentley TD, Teater P, Thomas A. et al. Immediate benefits realized following implementation of physician order entry at an academic medical center. J Am Med Inform Assoc 2002; 9: 529-539.
• 21 Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled trial of direct physician order entry: effects on physicians’ time utilization in ambulatory primary care internal medicine practices. J Am Med Inform Assoc 2001; 8: 361-369.
• 22 Pirnejad H, Niazkhani Z, van der Sijs H, Berg M, Bal R. Evaluation of the impact of a CPOE system on nurse-physician communication –a mixed method study. Methods Inf Med 2009; 48: 350-360.
• 23 Weir CR, Staggers N, Phansalkar S. The state of the evidence for computerized provider order entry: a systematic review and analysis of the quality of the literature. Int J Med Inform 2009; 78: 365-374.
• 24 Kaelber DC, Bates DW. Health information exchange and patient safety. J Biomed Inform 2007; 40 (Suppl. 06) S40-S45.
• 25 Staggers N, Weir C, Phansalkar S. Patient safety and health information technology: role of the electronic health record. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality; 2008. p. 91-133.
• 26 Van de Castle B, Kim J, Pedreira ML, Paiva A, Goossen W, Bates DW. Information technology and patient safety in nursing practice: an international perspective. Int J Med Inform 2004; 73: 607-614.
• 27 Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Stud Health Technol Inform 2009; 150: 532-536.
• 28 Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L. Relationship between medication errors and adverse drug events. J Gen Intern Med 1995; 10: 199-205.
• 29 Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA. et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991; 324: 377-384.
• 30 Institute of Medicine.. Preventing medication errors: Quality chasm series. Washington, DC: National Academy Press; 2006
• 31 DeYoung JL, Vanderkooi ME, Barletta JF. Effect of bar-code-assisted medication administration on medication error rates in an adult medical intensive care unit. Am J Health Syst Pharm 2009; 66 (Suppl. 12) 1110-1115.
• 32 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293: 1197-1203.
• 33 Shojania KG, Duncan BW, McDonald KM, Wachter RM. Safe but sound: patient safety meets evidence-based medicine. JAMA 2002; 288: 508-513.
• 34 Keohane CA, Bane AD, Featherstone E, Hayes J, Woolf S, Hurley A. et al. Quantifying nursing workflow in medication administration. J Nurs Adm 2008; 38: 19-26.
• 35 Nielsen J, Mack RL. Usability Inspection Methods. New York, NY: John Wiley & Sons; 1994
• 36 UsabilityNet.. Usability definitions. International Standards Organization or ISO 9241–11: Guidance on usability. 1998 [cited 2010 April 14]; Available from: http://www.usabilitynet.org/tools/r_international.htm#9241-11.
• 37 Sears A, Jacko JA. Human-computer interaction. Fundamentals. Boca Raton, FL: CRC Press; 2009
• 38 Rubin J, Chisnell D. Handbook of usability testing: How to plan, design, and conduct effective tests. 2^nd ed. Indianapolis, IN: Wiley Publishing, Inc; 2008
• 39 McDonnell C, Werner K, Wendel L. Electronic health record usability: Vendor practices and perspectives. AHRQ Publication No. 09(10)-0091–3-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2010
• 40 The University of Utah Health Care.. University hospitals & clinics: For patients. Salt Lake City: The University of Utah; n.d. [updated n.d.; cited 2010 August 18]; Available from: http://healthcare.utah.edu/hospitalpatients/.
• 41 Healthcare Information and Management Systems Society.. EMR adoption model. HIMSS Analytics;. 2010 [cited 2010 July 28]; Available from: http://www.himssanalytics.org/hc_providers/emr_adoption.asp.
• 42 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36: 23-30.
• 43 Nielsen J, Molich R. editors. Heuristic evaluation of user interfaces. SIGCHI Conference on Human Factors in Computing Systems: Empowering People. 1990. April 1-5 Seattle, WA: ACM.;
• 44 Nielsen J. Heuristic evaluation. In Nielsen J, Mack RL editors. Usability inspection methods. New York: John Wiley & Sons, Inc.; 1994
• 45 Shneiderman B, Plaisant C. Designing the user interface : strategies for effective human-computer interaction. 5^th ed. Boston: Addison-Wesley; 2010
• 46 Nielsen J. Usability engineering. Cambridge, MA: AP Professional; 1994
• 47 Dix A, Finlay JE, Abowd GD, Beale R. Human-computer interaction. 3^rd ed. Essex, England: Prentice Hall; 2004
• 48 Van der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc 2003; 10: 235-243.
• 49 Healthcare Information and Management Systems Society.. Defining and testing EMR usability: Principles and proposed methods of EMR usability evaluation and rating. Healthcare Information and Management Systems Society;. 2009 [cited 2010 July 28]; Available from: http://www.himss.org/content/files/HIMSS_DefiningandTestingEMRUsability.pdf.