A Heuristic Evaluation to Assess Use of After Visit Summaries for Supporting Continuity of Care

Abstract

Background Outpatient providers often do not receive discharge summaries from acute care providers prior to follow-up visits. These outpatient providers may use the after-visit summaries (AVS) that are given to patients to obtain clinical information. It is unclear how effectively AVS support care coordination between clinicians.

Objectives Goals for this effort include: (1) developing usability heuristics that may be applied both for assessment and to guide generation of medical documents in general, (2) conducting a heuristic evaluation to assess the use of AVS for communication between clinicians, and (3) providing recommendations for generating AVS that effectively support both patient/caregiver use and care coordination.

Methods We created a 17-item heuristic evaluation instrument for assessing usability of medical documents. Eight experts used the instrument to assess each of four simulated AVS. The simulations were created using examples from two hospitals and two pediatric patient cases developed by the National Institute of Standards and Technology.

Results Experts identified 224 unique usability problems ranging in severity from mild to catastrophic. Content issues (e.g., missing medical history, marital status of a 2-year-old) were rated as most severe, but widespread formatting and structural problems (e.g., inconsistent indentation, fonts, and headings; confusing ordering of information) were so distracting that they significantly reduced readers' ability to efficiently use the documents. Overall, issues in the AVS from Hospital 2 were more severe than those in the AVS from Hospital 1.

Conclusion The new instrument allowed for quick, inexpensive evaluations of AVS. Usability issues such as unnecessary information, poor organization, missing information, and inconsistent formatting make it hard for patients, caregivers, and clinicians to use the AVS. The heuristics in the new instrument may be used as guidance to adapt electronic health record systems so that they generate more useful and usable medical documents.

Keywords

testing and evaluation - electronic health records and systems - communication barriers - clinical documentation and communications - workarounds and unanticipated consequences - interfaces and usability

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by ECRI's Institutional Review Board.

• References

• 1 Yeaman B, Ko KJ, Alvarez del Castillo R. Care transitions in long-term care and acute care: health information exchange and readmission rates. Online J Issues Nurs 2015; 20 (03) 5
• 2 Moore C, Wisnivesky J, Williams S, McGinn T. Medical errors related to discontinuity of care from an inpatient to an outpatient setting. J Gen Intern Med 2003; 18 (08) 646-651
• 3 Hoyer EH, Odonkor CA, Bhatia SN, Leung C, Deutschendorf A, Brotman DJ. Association between days to complete inpatient discharge summaries with all-payer hospital readmissions in Maryland. J Hosp Med 2016; 11 (06) 393-400
• 4 Kind AJ, Smith MA. Documentation of mandated discharge summary components in transitions from acute to subacute care. AHRQ Patient Safety: New Directions and Alternative Approaches. Rockville, MD: Agency for Healthcare Research and Quality; 2008: 179-188
• 5 Jones CD, Cumbler E, Honigman B. , et al. Hospital to post-acute care facility transfers: identifying targets for information exchange quality improvement. J Am Med Dir Assoc 2017; 18 (01) 70-73
• 6 Coghlin DT, Leyenaar JK, Shen M. , et al. Pediatric discharge content: a multisite assessment of physician preferences and experiences. Hosp Pediatr 2014; 4 (01) 9-15
• 7 Solan LG, Sherman SN, DeBlasio D, Simmons JM. Communication challenges: a qualitative look at the relationship between pediatric hospitalists and primary care providers. Acad Pediatr 2016; 16 (05) 453-459
• 8 Shen MW, Hershey D, Bergert L, Mallory L, Fisher ES, Cooperberg D. Pediatric hospitalists collaborate to improve timeliness of discharge communication. Hosp Pediatr 2013; 3 (03) 258-265
• 9 The Office of the National Coordinator for Health Information Technology. Report to Congress April 2015: Report on Health Information Blocking. Washington, DC: Health IT; 2015
• 10 Adler-Milstein J, Jha AK. Health information exchange among U.S. hospitals: who's in, who's out, and why?. Healthc (Amst) 2014; 2 (01) 26-32
• 11 Iroju O, Soriyan A, Gambo I, Olaleke J. Interoperability in healthcare: benefits, challenges and resolutions. Int J Innovat Appl Stud 2013; 3 (01) 262-270
• 12 Khennou F, Khamlichi YI, Chaoui NEH. Evaluating electronic health records interoperability. International Conference on Information and Software Technologies; 2017:106–118
• 13 Marchibroda J. Health policy brief: interoperability. Health Aff 2014. Available at: https://www.healthaffairs.org/do/10.1377/hpb20140811.761828/full/ . Accessed July 22, 2018
• 14 Gorry T. Harnessing the power of the EMR to improve written communications. Presentation at “Transforming Health IT by Embedding Safety”; 2017. Partnership for Health IT Patient Safety Meeting, Plymouth Meeting, PA
• 15 ECRI Institute Patient Safety Organization deep dive: care coordination. Available at: https://www.ecri.org/components/PSOCore/Pages/DeepDive0915_CareCoordination.aspx?tab=2 . Accessed April 28, 2018
• 16 Bansard M, Clanet R, Raginel T. Proposal of standardised and logical templates for discharge letters and discharge summaries sent to general practitioners [in French]. Sante Publique 2017; 29 (01) 57-70
• 17 Federman AD, Sanchez-Munoz A, Jandorf L, Salmon C, Wolf MS, Kannry J. Patient and clinician perspectives on the outpatient after-visit summary: a qualitative study to inform improvements in visit summary design. J Am Med Inform Assoc 2017; 24 (e1): e61-e68
• 18 Lim SY, Jarvenpaa SL, Lanham HJ. Barriers to interorganizational knowledge transfer in post-hospital care transitions: review and directions for information systems research. J Manage Inf Syst 2015; 32 (03) 48-74
• 19 Newnham H, Barker A, Ritchie E, Hitchcock K, Gibbs H, Holton S. Discharge communication practices and healthcare provider and patient preferences, satisfaction and comprehension: a systematic review. Int J Qual Health Care 2017; 29 (06) 752-768
• 20 Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 1990:249–256
• 21 ISO. 9421–11: Ergonomic requirements for office work with visual display terminals (VDTs). Part 11–guidelines for specifying and measuring usability. Geneva: International Standards Organisation; 1997
• 22 Nielsen J. Reliability of severity estimates for usability problems found by heuristic evaluation. Proceeding 1992; 129-130
• 23 Nielsen J. 10 usability heuristics for user interface design. 1995. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ . Accessed April 28, 2018
• 24 Hermawati S, Lawson G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?. Appl Ergon 2016; 56: 34-51
• 25 Human Factors Engineering. [updated June 2017]. Available at: https://psnet.ahrq.gov/primers/primer/20 . Accessed April 28, 2018
• 26 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 27 Ratwani RM, Hettinger AZ, Fairbanks RJ. Barriers to comparing the usability of electronic health records. J Am Med Inform Assoc 2017; 24 (e1): e191-e193
• 28 Lowry SZ, Ramaiah M, Patterson ES. , et al. (NISTIR 7804–1) Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization. NIST Interagency/Internal Report (NISTIR)-7804–1; 2015
• 29 Middleton B, Bloomrosen M, Dente MA. , et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc 2013; 20 (e1): e2-e8
• 30 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2): 23-30
• 31 Shneiderman B. Designing the User Interface: Strategies for Effective Human-Computer Interaction. New Delhi, India: Pearson Education India; 2010
• 32 Kantner L, Shroyer R, Rosenbaum S. Structured heuristic evaluation of online documentation. IEEE International Professional Communication Conference; 2002:331–342
• 33 Simply put: a guide for creating easy-to-understand materials; 2009. Available at: https://www.cdc.gov/healthliteracy/pdf/Simply_Put.pdf . Accessed April 28, 2018
• 34 Readability Formulas. How to improve the readability of anything you write. Available at: http://readabilityformulas.com/articles/how-to-improve-the-readability-of-anything-you-write.php . Accessed April 28, 2018
• 35 Plain Language Action and Information Network. Federal plain language guidelines; 2011. Available at: http://www.plainlanguage.gov/howto/guidelines/FederalPLGuidelines/FederalPLGuidelines.pdf . Accessed April 28, 2018
• 36 Centers for Medicare and Medicaid Services. Toolkit for making written material clear and effective. Part; 2010. Available at: https://www.cms.gov/Outreach-and-Education/Outreach/WrittenMaterialsToolkit/index.html . Accessed April 28, 2018
• 37 Medline Plus. How to write easy-to-read health materials; 2013. Available at: https://medlineplus.gov/etr.html . Accessed April 28, 2018
• 38 Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res 2010; 468 (10) 2572-2580
• 39 Burke HB, Hoang A, Becher D. , et al. QNOTE: an instrument for measuring the quality of EHR clinical notes. J Am Med Inform Assoc 2014; 21 (05) 910-916
• 40 Lowry SZ, Quinn MT, Ramaiah M. , et al. (NISTIR 7865) A Human Factors Guide to Enhance EHR Usability of Critical User Interactions When Supporting Pediatric Patient Care. NIST Interagency/Internal Report (NISTIR)-7865; 2012
• 41 Tremoulet PD, McManus M, Baranov D. Rendering ICU data useful via formative evaluations of Trajectory, Tracking, and Trigger (T3TM) software. Proceedings 2017; 6 (01) 50-56
• 42 Sarzynski E, Hashmi H, Subramanian J. , et al. Opportunities to improve clinical summaries for patients at hospital discharge. BMJ Qual Saf 2017; 26 (05) 372-380
• 43 Nguyen OK, Kruger J, Greysen SR, Lyndon A, Goldman LE. Understanding how to improve collaboration between hospitals and primary care in postdischarge care transitions: a qualitative study of primary care leaders' perspectives. J Hosp Med 2014; 9 (11) 700-706
• 44 Ruth JL, Geskey JM, Shaffer ML, Bramley HP, Paul IM. Evaluating communication between pediatric primary care physicians and hospitalists. Clin Pediatr (Phila) 2011; 50 (10) 923-928
• 45 Unaka NI, Statile A, Haney J, Beck AF, Brady PW, Jerardi KE. Assessment of readability, understandability, and completeness of pediatric hospital medicine discharge instructions. J Hosp Med 2017; 12 (02) 98-101
• 46 Unnewehr M, Schaaf B, Marev R, Fitch J, Friederichs H. Optimizing the quality of hospital discharge summaries--a systematic review and practical tools. Postgrad Med 2015; 127 (06) 630-639
• 47 Albert W, Tullis T. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes Books; 2013
• 48 Dumas JS, Fox JE. Usability testing: current practice and future directions. In: Jacko J, Sears AL. (eds.) The Human-Computer Interaction Handbook. Boca Raton: CRC Press; 2012
• 49 Molich R, Jeffries R. Comparative expert reviews. Extended Abstracts; 2003: 1060-1061
• 50 Rubin J, Chisnell D. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Hoboken, NJ: John Wiley & Sons; 2008