From Usability Testing to Clinical Simulations: Bringing Context into the Design and Evaluation of Usable and Safe Health Information Technologies

Summary

Objectives: The objective of this paper is to explore human factors approaches to understanding the use of health information technology (HIT) by extending usability engineering approaches to include analysis of the impact of clinical context through use of clinical simulations.

Methods: Methods discussed are considered on a continuum from traditional laboratory-based usability testing to clinical simulations. Clinical simulations can be conducted in a simulation laboratory and they can also be conducted in real-world settings. The clinical simulation approach attempts to bring the dimension of clinical context into stronger focus. This involves testing of systems with representative users doing representative tasks, in representative settings/environments.

Results: Application of methods where realistic clinical scenarios are used to drive the study of users interacting with systems under realistic conditions and settings can lead to identification of problems and issues with systems that may not be detected using traditional usability engineering methods. In conducting such studies, careful consideration is needed in creating ecologically valid test scenarios. The evidence obtained from such evaluation can be used to improve both the usability and safety of HIT. In addition, recent work has shown that clinical simulations, in particular those conducted in-situ, can lead to considerable benefits when compared to the costs of running such studies.

Conclusion: In order to bring context of use into the testing of HIT, clinical simulation, involving observing representative users carrying out tasks in representative settings, holds considerable promise.

Keywords

Ussability - clinical simulation - patient safety - technology-induced error - health information systems

• References

• 1 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. The relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7- 8) 519-26.
• 2 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of CPOE in facilitating medication errors. J Am Med Inform Assoc 2005; 293 (10) 1197-203.
• 3 Magrabi F, Ong M-S, Runciman W, Coiera E. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010; 17: 663-70.
• 4 Magrabi F, Ong M, Runciman W, Coiera E. Using FDA reports to inform classification of HIT safety problems. J Am Med Inform Assoc 2012; 19: 45-53.
• 5 Samaranayake NR, Cheug STD, Chui WCM. Cheung BMY. Technology-related medication errors in a tertiary hospital: A 5-year analysis of reported medication incidents. Int J Med Inform 2012; 81: 828-33.
• 6 Carvalho CJ, Borycki EM, Kushniruk AK. Ensuring the safety of health information systems. Healthc Q 2009; 12: 49-54.
• 7 Rasmussen R, Kushniruk A. The long and twisting path: An efficiency evaluation of an electronic whiteboard system. Stud Health Technol Inform 2013; 183: 174-8.
• 8 Lilholt LH, Pedersen SS, Madsen I, Nielsen PH, Boye N, Andersen SK, Nohr C. Development of methods for usability evaluations of EHR systems. Stud Health Technol Inform 2006; 124: 341-6.
• 9 Borycki EM, Kushniruk AW, Kuwata S, Kannry J. Engineering the electronic health record for safety: A multi-video-based approach to diagnosing and preventing technology-induced error arising from usability problems. Stud Health Technol Inform 2011; 166: 197-205.
• 10 Beuscart-Zephir MC, Pelayo S, Anceaux F, Meaux J, Degroisse M, Degoulet P. Impact of CPOE on doctor-nurse cooperation for the medication ordering and administration process. Int J Med Inform 2005; 74 (7) 629-41.
• 11 Carvalho CJ, Borycki EM, Kushniruk AK. Ensuring the safety of health information systems. Healthc Q 2009; 12: 49-54.
• 12 Kushniruk A, Turner P. A framework for user involvement and context in the design and development of safe e-Health systems. Stud Health Technol Inform 2012; 180: 353-7.
• 13 Borycki EM, Lemieux-Charles L, Nagle L, Eysenbach G. Evaluating the impact of hybrid electronic-paper environments upon novice nurse information seeking. Methods Inf Med 2009; 48 (2) 137-43.
• 14 Pearce C, Shachak A, Kushniruk A, de Lusignan S. Usability: A critical dimension for assessing the quality of clinical systems. Inform Prim Care 2009; 17 (4) 195-8.
• 15 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (1) 56-76.
• 16 Ammenwerth E, Hackl WO, Bintzer K, Christoffersen TEH, Jensen S, Lawton K. et al. Simulation studies for the evaluation of health information technologies: experiences and results. HIM J 2012; 41 (2) 14-21.
• 17 Borycki E, Kushniruk A, Anderson J, Anderson M. Designing and integrating clinical and computer-based simulations in health informatics: From real-world to virtual reality. In: Cakaj S. editor. Modeling Simulation and Optimization: Focus on Applications. Vukovar, Croatia: In-Tech; 2010. p. 31-52.
• 18 Borycki EM, Keay L. Methods to assess the safety of health information systems. Healthc Q 2010; 13: 49-54.
• 19 Kushniruk AW, Borycki EM, Kuwata S, Watanabe H. Using a low-cost simulation for assessing the impact of a medication administration system on workflow. Stud Health Technol Inform 2008; 136: 567-72.
• 20 Zheng K, Padman R, Johnson MP, Diamond HS. Understanding technology adoption in clincal care: Clinician adoption behaviour of a point-of-care reminder system. IntJ Med Inform 2005; 74 (7) 535-43.
• 21 Kushniruk AW, Beuscart-Zephir MC, Grzes A, Borycki E, Watbled L, Kannry J. Increasing the safety of healthcare information systems through improved procurement: Toward a framework for selection of safe healthcare systems. Healthc Q 2010; 13: 53-8.
• 22 Marcilly R, Bernonville S, Riccioli C, Beuscart-Zephir MC. Patient safety-oriented usability testing: A pilot study. Stud Health Technol Inform 2012; 180: 368-72.
• 23 Nielsen J, Mack R. Usability inspection methods. New York: John Wiley; 1994
• 24 Sharp H, Preece J. Interaction design (3rd ed.). New York: Wiley; 2011
• 25 Kushniruk AW, Borycki EM. Low-cost usability engineering. Healthc Q 2006; 9 (4) 98-100 102.
• 26 Kushniruk AW, Borycki EM, Kannry J. Commercial versus in-situ usability testing of healthcare information systems: Towards “public” usability tesing in healthcare organizations. Stud Health Technol Inform 2013; 183: 157-61.
• 27 Baylis T, Kushniruk AW, Borycki EM. Low-cost usability for health information systems: Is it worth the effort?. Stud Health Technol Inform 2012; 180: 363-7.
• 28 Hall SA, Kushniruk AW, Borycki EM. Usability analysis of the tele-nursing call management software at HealthLink BC. Stud Health Technol Inform 2011; 164: 208-12.
• 29 Li AC, Kannry JL, Kushniruk A, Chrimes D, McGinn TG, Edonyabo D, Mann DM. Integrating usability testing and think-aloud protocol analysis with "near live" clincal simulations in evaluting clinical decision support. Int J Med Inform 2012; 81 (11) 761-21.
• 30 Kushniruk AW, Borycki EM, Kuwata S, Kannry J. Emerging approaches to usability evaluation of health information systems: Towards in-situ analysis of complex healthcare systems and envioronments. Stud Health Technol Inform 2011; 169: 915-9.
• 31 Rasmussen SL, Lyng KM, Jensen S. Achieving IT-supported standardized mursing documentation through participatory design. Stud Health Technol Inform 2012; 180: 1055-9.
• 32 Ammenwerth E, Hackl WO, Jensen S, Lawton K, Riedmann D, Jung M. Impact evaluation of innovative technology: estimating the impact of the PSIP solutions. Stud Health Technol Inform 2011; 166: 227-33.
• 33 Kjeldskov J, Skov MB, Stage J. Instant data analysis: conducting usability evaluations in a day. NordiCHI '04 Proceedings of the third Nordic conference on Human-computer interaction, ACM. 2004
• 34 Jensen S, Kanstrup AM, Nohr C. User driven development and high fidelity testing. Proceedings Eleventh Danish HCI Research Symposium. 2011
• 35 Jensen S, Vingtoft S, Nohr C. Benefits of a clinical planning and coordination module: A simulation study. Stud Health Technol Inform 2013; 183: 220-4.
• 36 Lawton K, Bintzer K, Skjoet P, Jensen S. Lessons learned from conducting a high fidelity simulation test in health IT. Stud Health Technol Inform 2011; 166: 217-26.
• 37 Interaction design. Ecological validity. [Internet]. Accessed from: http://www.interaction-design.org/encyclopedia/ecological_validity.html
• 38 Elmes DG, Kantowitz BH, Roediger HL. Research methods in psychology. New York: Wadsworth Publishing Company; 2011
• 39 Sanderson P. Designing and evaluating healthcare ICT innovation: A cognitive engineering view. Proceedings of the Third International Conference on Information Technology in Healthcare-Socio-technical Systems. Sydney: 28-30 August 2007
• 40 Carayon P. Human factors and ergonomics in health care and patient safety (Second edition). New York: CRC Press; 2012
• 41 Jaspers M. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform 2009; 78: 340-53.