Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

Abstract

Background Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups.

Objectives We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard.

Methods An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data.

Results The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models.

Conclusion We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

Keywords

data visualization - interface and usability - data quality - data analysis - dashboard - testing and evaluation - human–computer interaction - mixed (methodologies)

• References

• 1 Adler-Milstein J, Jha AK. HITECH Act drove large gains in hospital electronic health record adoption. Health Aff (Millwood) 2017; 36 (08) 1416-1422
• 2 Non-federal Acute Care Hospital Health IT Adoption and Use. Available at: https://dashboard.healthit.gov/apps/hospital-health-it-adoption.php . Accessed April 15, 2019
• 3 Luo G. MLBCD: a machine learning tool for big clinical data. Health Inf Sci Syst 2015; 3: 3
• 4 Malykh VL, Rudetskiy SV. Approaches to medical decision-making based on big clinical data. J Healthc Eng 2018; 2018: 3917659
• 5 Kahn MG, Eliason BB, Bathurst J. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proc AMIA Symp 2010; 2010: 356-360
• 6 Zhang Y, Sun W, Gutchell EM. , et al. QAIT: a quality assurance issue tracking tool to facilitate the improvement of clinical data quality. Comput Methods Programs Biomed 2013; 109 (01) 86-91
• 7 Caban JJ, Gotz D. Visual analytics in healthcare--opportunities and research challenges. J Am Med Inform Assoc 2015; 22 (02) 260-262
• 8 Gotz D, Borland D. Data-driven healthcare: challenges and opportunities for interactive visualization. IEEE Comput Graph Appl 2016; 36 (03) 90-96
• 9 Bauer DT, Guerlain S, Brown PJ. The design and evaluation of a graphical display for laboratory data. J Am Med Inform Assoc 2010; 17 (04) 416-424
• 10 Plaisant C, Wu J, Hettinger AZ, Powsner S, Shneiderman B. Novel user interface design for medication reconciliation: an evaluation of Twinlist. J Am Med Inform Assoc 2015; 22 (02) 340-349
• 11 Mandalika VBH, Chernoglazov AI, Billinghurst M. , et al. A hybrid 2D/3D user interface for radiological diagnosis. J Digit Imaging 2018; 31 (01) 56-73
• 12 Quick Stats. These data visualizations of key data and statistics provide quick access to the latest facts and figures about health IT. Published February 6, 2019. Available at: https://dashboard.healthit.gov/quickstats/quickstats.php . Accessed April 15, 2019
• 13 Huber TC, Krishnaraj A, Monaghan D, Gaskin CM. Developing an interactive data visualization tool to assess the impact of decision support on clinical operations. J Digit Imaging 2018; 31 (05) 640-645
• 14 Romero-Brufau S, Kostandy P, Maass KL. , et al. Development of data integration and visualization tools for the Department of Radiology to display operational and strategic metrics. AMIA Annu Symp Proc AMIA Symp 2018; 2018: 942-951
• 15 Fletcher GS, Aaronson BA, White AA, Julka R. Effect of a real-time electronic dashboard on a rapid response system. J Med Syst 2017; 42 (01) 5
• 16 Radhakrishnan K, Monsen KA, Bae S-H, Zhang W. ; Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
• 17 Simpao AF, Ahumada LM, Larru Martinez B. , et al. Design and implementation of a visual analytics electronic antibiogram within an electronic health record system at a tertiary pediatric hospital. Appl Clin Inform 2018; 9 (01) 37-45
• 18 Hester G, Lang T, Madsen L, Tambyraja R, Zenker P. Timely data for targeted quality improvement interventions: use of a visual analytics dashboard for bronchiolitis. Appl Clin Inform 2019; 10 (01) 168-174
• 19 Karami M, Safdari R. From information management to information visualization: development of radiology dashboards. Appl Clin Inform 2016; 7 (02) 308-329
• 20 Federici S, Borsci S. Usability evaluation: models, methods, and applications. International encyclopedia of rehabilitation. Center for International rehabilitation research information and exchange (CIRRIE); 2010: 1-17
• 21 Wu DTY, Chen AT, Manning JD. , et al. Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation. J Am Med Inform Assoc 2019; 26 (04) 314-323
• 22 Norman DA. The Design of Everyday Things. Revised and expanded edition. New York, New York: Basic Books; 2013
• 23 Mental Models. Available at: https://www.nngroup.com/articles/mental-models/ . Accessed April 15, 2019
• 24 Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002; 9 (04) 320-330
• 25 Kurzhals K, Hlawatsch M, Seeger C, Weiskopf D. Visual analytics for mobile eye tracking. IEEE Trans Vis Comput Graph 2017; 23 (01) 301-310
• 26 Ploderer B, Brown R, Seng LSD, Lazzarini PA, van Netten JJ. Promoting self-care of diabetic foot ulcers through a mobile phone app: user-centered design and evaluation. JMIR Diabetes 2018; 3 (04) e10105
• 27 Anoshin D, Rana H, Ma N. Mastering Business Intelligence with MicroStrategy Build World-Class Enterprise Business Intelligence Solutions with MicroStrategy 10. Birmingham, UK: Packt Publishing; 2016
• 28 The Meaning Behind STAT Scores & Categories. Available at: https://www.luriechildrens.org/en/specialties-conditions/heart-center/volumes-outcomes/the-meaning-behind-stat-scores-categories/ . Accessed August 5, 2019
• 29 van Lieshout R, Pisters MF, Vanwanseele B, de Bie RA, Wouters EJ, Stukstette MJ. Biofeedback in partial weight bearing: usability of two different devices from a patient's and physical therapist's perspective. PLoS One 2016; 11 (10) e0165199
• 30 Hamm J, Money AG, Atwal A, Ghinea G. Mobile three-dimensional visualisation technologies for clinician-led fall prevention assessments. Health Informatics J 2019; 25 (03) 788-810
• 31 Nitsch M, Adamcik T, Kuso S, Zeiler M, Waldherr K. Usability and engagement evaluation of an unguided online program for promoting a healthy lifestyle and reducing the risk for eating disorders and obesity in the school setting. Nutrients 2019; 11 (04) E713
• 32 Hirschmann J, Sedlmayr B, Zierk J. , et al. Evaluation of an interactive visualization tool for the interpretation of pediatric laboratory test results. Stud Health Technol Inform 2017; 243: 207-211
• 33 He X, Zhang R, Rizvi R. , et al. ALOHA: developing an interactive graph-based visualization for dietary supplement knowledge graph through user-centered design. BMC Med Inform Decis Mak 2019; 19 (Suppl. 04) 150
• 34 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
• 35 Brooke J. System Usability Scale (SUS): A Quick-And-Dirty Method of System Evaluation User Information. Read UK Digit Equip Co Ltd.; 1986: 43
• 36 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4 (03) 114-123
• 37 Coe AM, Ueng W, Vargas JM. , et al. Usability testing of a web-based decision aid for breast cancer risk assessment among multi-ethnic women. AMIA Annu Symp Proc AMIA Symp 2016; 2016: 411-420
• 38 Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc 2016; 23 (01) 5-11
• 39 Maguire M, Delahunt B. Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. AISHE-J Irel J Teach Learn High Educ 2017 ;9(3). Available at: http://ojs.aishe.org/index.php/aishe-j/article/view/335 . Accessed September 15, 2019
• 40 Wenger MJ, Spyridakis JH. The relevance of reliability and validity to usability testing. IEEE Trans Prof Commun 1989; 32 (04) 265-271
• 41 Kessner M, Wood J, Dillon RF, West RL. On the reliability of usability testing. In: CHI '01 Extended Abstracts on Human Factors in Computing Systems - CHI '01. Seattle, Washington: ACM Press; 2001: 97