Download PDF
CC BY-NC-ND 4.0 · Appl Clin Inform 2020; 11(04): 528-534
DOI: 10.1055/s-0040-1714693
Case Report

Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process

Regina Richter Lagha
1   Veterans Affairs Greater Los Angeles Medical Center, Geriatric Research Education and Clinical Center, Los Angeles, California, United States
,
Zachary Burningham
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Brian C. Sauer
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Jianwei Leng
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Celena Peters
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Tina Huynh
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Shardool Patel
2   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
,
Ahmad S. Halwani
3   Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States
4   Division of Epidemiology, Department of Hematology, University of Utah, Salt Lake City, Utah, United States
,
B. Josea Kramer
1   Veterans Affairs Greater Los Angeles Medical Center, Geriatric Research Education and Clinical Center, Los Angeles, California, United States
5   Division of Geriatric Medicine, David Geffen School of Medicine, University of California at Los Angeles, Los Angeles, California, United States
› Author Affiliations
› Further Information
   Permissions and Reprints

Abstract

Background With the increased usage of dashboard reporting systems to monitor and track patient panels by clinical users, developers must ensure that the information displays they produce are accurate and intuitive. When evaluating usability of a clinical dashboard among potential end users, developers oftentimes rely on methods such as questionnaires as opposed to other, more time-intensive strategies that incorporate direct observation.

Objectives Prior to release of the potentially inappropriate medication (PIM) clinical dashboard, designed to facilitate completion of a quality improvement project by clinician scholars enrolled in the Veterans Affairs (VA) workforce development Geriatric Scholars Program (GSP), we evaluated the usability of the system. This article describes the process of usability testing a dashboard reporting system with clinicians using direct observation and think-aloud moderating techniques.

Methods We developed a structured interview protocol that combines virtual observation, think-aloud moderating techniques, and retrospective questioning of the overall user experience, including use of the System Usability Scale (SUS). Thematic analysis was used to analyze field notes from the interviews of three GSP alumni.

Results Our structured approach to usability testing identified specific functional problems with the dashboard reporting system that were missed by results from the SUS. Usability testing lead to overall improvements in the intuitive use of the system, increased data transparency, and clarification of the dashboard's purpose.

Conclusion Reliance solely on questionnaires and surveys at the end stages of dashboard development can mask potential functional problems that will impede proper usage and lead to misinterpretation of results. A structured approach to usability testing in the developmental phase is an important tool for developers of clinician friendly systems for displaying easily digested information and tracking outcomes for the purpose of quality improvement.

Keywords

quality improvement - informatics - user–computer interface - inappropriate medication - decision support systems

Protection of Human and Animal Subjects

This study was characterized as a nonresearch operational activity and not human subject research.




Publication History

Received: 24 March 2020

Accepted: 21 June 2020

Article published online:
12 August 2020

© 2020. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial-License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Georg Thieme Verlag KG
Stuttgart · New York

 

  • References

  • 1 Ivers NM, Barrett J. Using report cards and dashboards to drive quality improvement: lessons learnt and lessons still to learn. BMJ Qual Saf 2018; 27 (06) 417-420
    CrossrefPubMedGoogle Scholar
  • 2 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
    CrossrefPubMedGoogle Scholar
  • 3 Egan M. Clinical dashboards: impact on workflow, care quality, and patient safety. Crit Care Nurs Q 2006; 29 (04) 354-361
    CrossrefPubMedGoogle Scholar
  • 4 Ghazisaeidi M, Safdari R, Torabi M, Mirzaee M, Farzi J, Goodini A. Development of performance dashboards in healthcare sector: key practical issues. Acta Inform Med 2015; 23 (05) 317-321
    CrossrefPubMedGoogle Scholar
  • 5 Sujansky W. Heterogeneous database integration in biomedicine. J Biomed Inform 2001; 34 (04) 285-298
    CrossrefPubMedGoogle Scholar
  • 6 Armijo D, McDonnell C, Werner K. . Electronic Health Record Usability: Evaluation and Use Case Framework. Rockville, MD: Agency for Healthcare Research and Quality; October 2009 ; AHRQ Publication No. 09(10)-0091–1-EF
    PubMed
  • 7 Bastien JMC. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010; 79 (04) e18-e23
    CrossrefPubMedGoogle Scholar
  • 8 Dolan JE, Lonsdale H, Ahumada LM. , et al. Quality initiative using theory of change and visual analytics to improve controlled substance documentation discrepancies in the operating room. Appl Clin Inform 2019; 10 (03) 543-551
    Article in Thieme ConnectPubMedGoogle Scholar
  • 9 Dunn S, Sprague AE, Grimshaw JM. , et al. A mixed methods evaluation of the maternal-newborn dashboard in Ontario: dashboard attributes, contextual factors, and facilitators and barriers to use: a study protocol. Implement Sci 2016; 11: 59
    CrossrefPubMedGoogle Scholar
  • 10 Lee K, Jung SY, Hwang H. , et al. A novel concept for integrating and delivering health information using a comprehensive digital dashboard: an analysis of healthcare professionals' intention to adopt a new system and the trend of its real usage. Int J Med Inform 2017; 97: 98-108
    CrossrefPubMedGoogle Scholar
  • 11 Mlaver E, Schnipper JL, Boxer RB. , et al. User-centered collaborative design and development of an inpatient safety dashboard. Jt Comm J Qual Patient Saf 2017; 43 (12) 676-685
    PubMedGoogle Scholar
  • 12 Bhutkar G, Konkani A, Katre D, Ray GG. A review: healthcare usability evaluation methods. Biomed Instrum Technol 2013; 45-53
    CrossrefPubMedGoogle Scholar
  • 13 U.S. Department of Health & Human Services. What & Why of Usability. Available at: https://www.usability.gov/how-to-and-tools/methods/usability-testing.html . Accessed October 12, 2018
    PubMed
  • 14 Kramer BJ, Creekmur B, Howe JL. , et al. Veterans Affairs Geriatric Scholars Program: enhancing existing primary care clinician skills in caring for older veterans. J Am Geriatr Soc 2016; 64 (11) 2343-2348
    CrossrefPubMedGoogle Scholar
  • 15 Burningham Z, Chen W, Sauer BC. , et al. VA Geriatric Scholars Program's impact on prescribing potentially inappropriate medications. Am J Manag Care 2019; 25 (09) 425-430
    PubMedGoogle Scholar
  • 16 Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf 2014; 23 (04) 290-298
    CrossrefPubMedGoogle Scholar
  • 17 2019 American Geriatrics Society Beers Criteria® Update Expert Panel. American Geriatrics Society Health Services Research & Development (HSR&D) VIReC Clinical Informatics Seminar Series. VA Geriatric Scholars Quality Improvement Dashboards: Leveraging Informatics to Facilitate Change, 6/18/2019. Available at: https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3643 . Accessed July 6, 2020
    PubMed
  • 18 2019 American Geriatrics Society Beers Criteria® Update Expert Panel. American Geriatrics Society 2019 Updated AGS Beers Criteria® for Potentially Inappropriate Medication Use in Older Adults. J Am Geriatr Soc 2019; 67: 674-694
    CrossrefPubMedGoogle Scholar
  • 19 Brooke J. SUS: A “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL. , eds. Usability Evaluation in Industry. London: Taylor and Francis; 1996: 189-194
    Google Scholar
  • 20 Research-Based Web Design & Usability Guidelines. U.S. Department of Health and Human Services. 2006
  • 21 Ericsson K, Simon H. Protocol Analysis: Verbal Reports as Data, 2nd ed. Boston: MIT Press; 1993
    CrossrefGoogle Scholar
  • 22 Romano Bergstrom J. . Moderating Usability Tests. U.S. Department of Health and Human Services Usability website. Available at: https://www.usability.gov/get-involved/blog/2013/04/moderating-usability-tests.html . April 2, 2013 . Accessed May 16, 2020
    PubMed
  • 23 Castleberry A, Nolen A. Thematic analysis of qualitative research data: Is it as easy as it sounds?. Curr Pharm Teach Learn 2018; 10 (06) 807-815
    CrossrefPubMedGoogle Scholar
  • 24 Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017; 16 (01) 1-13
    CrossrefPubMedGoogle Scholar
  • 25 Ratwani RM, Reider J, Singh H. A decade of health information technology usability challenges and the path forward. JAMA 2019; 321 (08) 743-744
    CrossrefPubMedGoogle Scholar
  • 26 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
    CrossrefPubMedGoogle Scholar
  • 27 Johnson CM, Johnston D, Crowley PK. , et al. EHR Usability Toolkit: A Background Report on Usability and Electronic Health Records (Prepared by Westat under Contract No. HHSA 290- 2009–00023I). AHRQ Publication No. 11–0084-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2011
    PubMed
  • 28 Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95-104
    CrossrefPubMedGoogle Scholar
  • 29 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
    Article in Thieme ConnectPubMedGoogle Scholar
  • 30 Dowding D, Merrill JA, Barrón Y, Onorato N, Jonas K, Russell D. Usability evaluation of a dashboard for home care nurses. Comput Inform Nurs 2019; 37 (01) 11-19
    CrossrefPubMedGoogle Scholar
  • 31 Voluntary Guidelines for the Design of Clinical Decision Support Software to Assure the Central Role of Healthcare Professionals in Clinical Decision-Making [Internet].Washington, DC: CDS Coalition; 2017. Available at: http://cdscoalition.org/wp-content/uploads/2017/04/CDS-3060-Guidelines-032717-with-memo.pdf . Accessed July 6, 2020
    PubMed
  • 32 Singh H, Meyer AN, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf 2014; 23 (09) 727-731
    CrossrefPubMedGoogle Scholar
  • 33 Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt. BMJ 2003; 326 (7394): 860-863
    CrossrefPubMedGoogle Scholar
  • 34 Boren MT, Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Prof Commun 2000; 43 (03) 261-278
    CrossrefPubMedGoogle Scholar
  • 35 Nisbett RE, Wilson TD. Telling more than we know: verbal reports on mental processes. Psychol Rev 1977; 84 (03) 231-259
    CrossrefPubMedGoogle Scholar