Download PDF
Appl Clin Inform 2018; 09(03): 511-518
DOI: 10.1055/s-0038-1666842
Research Article
Georg Thieme Verlag KG Stuttgart · New York

The Development of Heuristics for Evaluation of Dashboard Visualizations

Dawn Dowding
1   Division of Nursing, Midwifery and Social Work, School of Health Sciences, University of Manchester, Manchester, United Kingdom
,
Jacqueline A. Merrill
2   Department of Biomedical Informatics, Columbia University, New York, New York, United States
3   School of Nursing, Columbia University, New York, New York, United States
› Author Affiliations Funding The research reported here was supported by the Agency for Healthcare Research and Quality (U.S.) under award number R21HS023855. The content is solely the responsibility of the authors and does not necessarily represent the official viess of the Agency for Healthcare Research and Quality.
Further Information

Publication History

15 March 2018

21 May 2018

Publication Date:
11 July 2018 (online)

   Permissions and Reprints

Abstract

Background Heuristic evaluation is used in human–computer interaction studies to assess the usability of information systems. Nielsen's widely used heuristics, first developed in 1990, are appropriate for general usability but do not specifically address usability in systems that produce information visualizations.

Objective This article develops a heuristic evaluation checklist that can be used to evaluate systems that produce information visualizations. Principles from Nielsen's heuristics were combined with heuristic principles developed by prior researchers specifically to evaluate information visualization.

Methods We used nominal group technique to determine an appropriate final set. The combined existing usability principles and associated factors were distributed via email to a group of 12 informatics experts from a range of health care disciplines. Respondents were asked to rate each factor on its importance as an evaluation heuristic for visualization systems on a scale from 1 (definitely don't include) to 10 (definitely include). The distribution of scores for each item were calculated. A median score of ≥8 represented consensus for inclusion in the final checklist.

Results Ten of 12 experts responded with rankings and written comments. The final checklist consists of 10 usability principles (7 general and 3 specific to information visualization) substantiated by 49 usability factors. Three nursing informatics experts then used the checklist to evaluate a vital sign dashboard developed for home care nurses, using a task list designed to explore the full functionality of the dashboard. The experts used the checklist without difficulty, and indicated that it covered all major usability problems encountered during task completion.

Conclusion The growing capacity to generate and electronically process health data suggests that data visualization will be increasingly important. A checklist of usability heuristics for evaluating information visualization systems can contribute to assuring high quality in electronic data systems developed for health care.

Keywords

requirements analysis and design - clinical decision support - home health - dashboard - methodologies

Protection of Human and Animal Subjects

The study protocol was reviewed and approved by the Institutional Review Boards at Columbia University and the Visiting Nurse Service of New York.


Supplementary Material

 

  • References

  • 1 Andrews K. Evaluation comes in many guises. In: Proceedings of the 2008 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization (BELIV, 2008); Florence, Italy. New York: ACM; 2008
    PubMedGoogle Scholar
  • 2 Hermawati S, Lawson G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?. Appl Ergon 2016; 56: 34-51
    CrossrefPubMedGoogle Scholar
  • 3 10 usability heuristics for interface design. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ . Accessed June 10, 2018
    PubMed
  • 4 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In: CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Seattle, Washington. New York: ACM; 1990 :249–256
    PubMedGoogle Scholar
  • 5 Pierotti D. Heuristic Evaluation - A System Checklist. Available at: ftp://ftp.cs.uregina.ca/pub/class/305/lab2/example-he.html . Accessed June 10, 2018
    PubMedGoogle Scholar
  • 6 Tarrell A, Forsell C, Fruhling AL, Grinstein G, Borgo R, Scholtz J. Toward Visualization-Specific Heuristic Evaluation. In: BELIV '14 Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization Paris, France. New York: ACM; 2014 :110–117
    PubMedGoogle Scholar
  • 7 Väätäjä H, Varsaluoma J, Heimonen T. , et al. Information visualization heuristics in practical expert evaluation. In: BELIV '16 Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization Baltimore, MD. New York: ACM; 2016 :36–43
    PubMedGoogle Scholar
  • 8 Gonzalez-Holland E, Whitmer D, Moralez L, Mouloua M. Examination of the use of Nielsen's 10 usability heuristics & outlooks for the future. Proc Hum Factors Ergon Soc Annu Meet 2017; 61 (01) 1472-1475
    CrossrefPubMedGoogle Scholar
  • 9 Forsell C, Johansson J. An heuristic set for evaluation in information visualization. In: AVI '10 Proceedings of the International Conference on Advanced Visual Interfaces Roma, Italy. New York: ACM; 2010 :199–206
    PubMedGoogle Scholar
  • 10 Zuk T, Schlesier L, Neumann P, Hancock MS, Carpendale S. Heuristics for information visualization evaluation. In: BELIV '06 Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization Venice, Italy. New York: ACM; 2006 :1–6
    PubMedGoogle Scholar
  • 11 Freitas C, Luzzardi P, Cava R, Winckler M, Pimenta MS, Nedel L. Evaluating usability of information visualization techniques. In: Proceedings of the 5th Symposium on Human Factors in Computer Systems, IHC 2002; Brazil. Brazil: Brazilian Computer Society; 2002 :40–51
    PubMedGoogle Scholar
  • 12 Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Scott Foresman; 1975
    Google Scholar
  • 13 Delbecq AL, Van de Ven AH. A group process model for problem identification and program planning. J Appl Behav Sci 1971; 7 (04) 466-492
    CrossrefPubMedGoogle Scholar
  • 14 Amar R, Stasko J. A knowledge task-based framework for design and evaluation of information visualizations. In: Proceedings of IEEE Symposium on Information Visualization, 2004 INFOVIS 2004; Austin, TX, USA. Washington, DC: IEEE Computer Society; 2004
    PubMedGoogle Scholar
  • 15 Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic quality of interactive system. Behav Inf Technol 1997; 16 (4–5): 220-231
    CrossrefPubMedGoogle Scholar
  • 16 Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the IEEE Symposium on Visual Languages; Boulder, CO, USA. Washington, DC: IEEE Computer Society Press; 1996 :336–343
    PubMedGoogle Scholar
  • 17 Nielsen J. Enhancing the explanatory power of usability heuristics. In: CHI '94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: April 24–28, 1994; Boston, MA. New York, NY: ACM; 1994 :152–158
    PubMedGoogle Scholar
  • 18 Zuk T, Carpendale S. Theoretical analysis of uncertainty visualizations. Proc. SPIE 6060, Visualization and Data Analysis 2006, 606007. Doi: 10.1117/12.643631
    PubMedGoogle Scholar
  • 19 Stone JV. Principles of Neural Information Theory: A Tutorial Introduction. Sheffield, UK: Sebtel Press; 2016
    Google Scholar
  • 20 Dowding DW, Russell D, Onorato N, Merrill JA. Technology solutions to support care continuity in home care: a focus group study. J Healthc Qual 2017
    PubMedGoogle Scholar
  • 21 Docherty SL, Vorderstrasse A, Brandon D, Johnson C. Visualization of multidimensional data in nursing science. West J Nurs Res 2016; 39 (01) 112-126
    PubMedGoogle Scholar
  • 22 Schumacher R, Lowry S. NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Gaithersburg, MD: National Institute of Standards and Technology; 2010
    Google Scholar
  • 23 Rind A, Wang T, Aigner W. , et al. Interactive information visualization to explore and query electronic health records. Foundations Trends Human Comp Interact 2013; 5 (03) 207-298
    PubMedGoogle Scholar
  • 24 Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Extracting insights from electronic health records: case studies, a visual analytics process model, and design recommendations. J Med Syst 2011; 35 (05) 1135-1152
    CrossrefPubMedGoogle Scholar
  • 25 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
    CrossrefPubMedGoogle Scholar
  • 26 Shneiderman B, Plaisant C, Hesse B. Improving healthcare with interactive visualization. Computer 2013; 46 (05) 58-66
    CrossrefPubMedGoogle Scholar
  • 27 National Library of Medicine (U.S.). Board of Regents: a platform for biomedical discovery and data-powered health: National Library of Medicine strategic plan 2017–202. In: NIH Publication. Bethesda, MD: National Institutes of Health, National Library of Medicine; 2017
    Google Scholar