Download PDF
CC BY-NC-ND 4.0 · Appl Clin Inform 2022; 13(01): 67-79
DOI: 10.1055/s-0041-1740919
Research Article

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation

Irina Sinabell
1   Department of Biomedical Computer Science and Mechatronics, Institute of Medical Informatics, UMIT, Private University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
,
Elske Ammenwerth
1   Department of Biomedical Computer Science and Mechatronics, Institute of Medical Informatics, UMIT, Private University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
› Author Affiliations
› Further Information
   Permissions and Reprints

Abstract

Background Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.

Objective Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.

Methods The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.

Results Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.

Conclusion We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Keywords

telemedicine and telehealth - clinical research informatics - clinical information systems - human–computer interaction - hospital information systems

Protection of Human and Animal Subjects

Ethical approval was not required for this study.


Supplementary Material



Publication History

Received: 24 May 2021

Accepted: 20 October 2021

Article published online:
09 March 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Lau F, Kuziemsky CE. eds. Handbook of eHealth Evaluation: An Evidence-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016
    Google Scholar
  • 2 International Telecommunication Union (ITU). Standardization in e-health. Accessed August 9, 2021 at: https://www.itu.int/itunews/issue/2003/06/standardization.html
    Google Scholar
  • 3 World Health Organization (WHO). eHealth. Accessed August 9, 2021 at: https://www.euro.who.int__data/assets/pdf_file/0010/261694/6.-eHealth,-Factsheet-for-European-Parliament.pdf
    Google Scholar
  • 4 Bockhacker M, Syrek H, Elstermann von Elster M, Schmitt S, Roehl H. Evaluating usability of a touchless image viewer in the operating room. Appl Clin Inform 2020; 11 (01) 88-94
    Article in Thieme ConnectPubMedGoogle Scholar
  • 5 Richter Lagha R, Burningham Z, Sauer BC. et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020; 11 (04) 528-534
    Article in Thieme ConnectPubMedGoogle Scholar
  • 6 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Hron JD, Parsons CR, Williams LA, Harper MB, Bourgeois FC. Rapid implementation of an inpatient telehealth program during the COVID-19 pandemic. Appl Clin Inform 2020; 11 (03) 452-459
    Article in Thieme ConnectPubMedGoogle Scholar
  • 8 Rödle W, Wimmer S, Zahn J. et al. User-centered development of an online platform for drug dosing recommendations in pediatrics. Appl Clin Inform 2019; 10 (04) 570-579
    Article in Thieme ConnectPubMedGoogle Scholar
  • 9 Farzandipour M, Nabovati E, Heidarzadeh Arani M, Akbari H, Sharif R, Anvari S. Enhancing asthma patients' self-management through smartphone-based application: design, usability evaluation, and educational intervention. Appl Clin Inform 2019; 10 (05) 870-878
    Article in Thieme ConnectPubMedGoogle Scholar
  • 10 Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems-a literature survey. Int J Med Inform 2016; 93: 57-69
    CrossrefPubMedGoogle Scholar
  • 11 Kushniruk AW, Borycki EM. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc Q 2006; 9 (04) 98-102
    PubMedGoogle Scholar
  • 12 Ali H, Cole A, Panos G. Transforming patient hospital experience through smart technologies. In: Proceedings of the International Conference on Human-Computer Interaction; HCII '20. Copenhagen, Denmark: Springer International Publishing; 2020: 203-215
    Google Scholar
  • 13 Coppersmith NA, Sarkar IN, Chen ES. Quality informatics: the convergence of healthcare data, analytics, and clinical excellence. Appl Clin Inform 2019; 10 (02) 272-277
    Article in Thieme ConnectPubMedGoogle Scholar
  • 14 Kushniruk A, Borycki E. Low-cost rapid usability testing: its application in both product development and system implementation. Stud Health Technol Inform 2017; 234: 195-200
    PubMedGoogle Scholar
  • 15 Price M, Weber J, Bellwood P, Diemert S, Habibi R. Evaluation of eHealth system usability and safety. In: Handbook of EHealth Evaluation: An Evidece-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016: 337-350
    Google Scholar
  • 16 Kushniruk AW, Borycki EM. Integrating low-cost rapid usability testing into agile system development of healthcare IT: a methodological perspective. Stud Health Technol Inform 2015; 210: 200-204
    PubMedGoogle Scholar
  • 17 Marcilly R, Schiro J, Beuscart-Zéphir MC, Magrabi F. Building usability knowledge for health information technology: a usability-oriented analysis of incident reports. Appl Clin Inform 2019; 10 (03) 395-408
    Article in Thieme ConnectPubMedGoogle Scholar
  • 18 Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform 2019; 128: 24-31
    CrossrefPubMedGoogle Scholar
  • 19 International Organization for Standardization (ISO). Ergonomics of human-system interaction - Part 11: usability: definitions and concepts. Accessed July 26, 2021 at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
    Google Scholar
  • 20 Paelke V, Röcker C. User interfaces for cyber-physical systems: challenges and possible approaches. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 75-85
    Google Scholar
  • 21 Bastien JM. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010; 79 (04) e18-e23
    CrossrefPubMedGoogle Scholar
  • 22 Speicher M, Both A, Gaedke M. INUIT: the interface usability instrument. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 256-268
    Google Scholar
  • 23 Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95-104
    CrossrefPubMedGoogle Scholar
  • 24 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
    CrossrefPubMedGoogle Scholar
  • 25 Thompson KE, Rozanski EP, Haake AR. Here, there, anywhere: remote usability testing that works. In: Proceedings of the 5th Conference on Information Technology Education; CITC5 '04. Salt Lake City, UT, USA: ACM; 2004: 132-137
    Google Scholar
  • 26 Wysocki T, Pierce J, Caldwell C. et al. A web-based coping intervention by and for parents of very young children with type 1 diabetes: user-centered design. JMIR Diabetes 2018; 3 (04) e16
    CrossrefPubMedGoogle Scholar
  • 27 González Sánchez JL, Padilla Zea N, Gutiérrez FL. From usability to playability: introduction to player-centred video game development process. In: Proceedings of the International Conference on Human Centered Design; HCD '09. San Diego, CA, USA: Springer; 2009. ;5619: 65-74
    Google Scholar
  • 28 Dubey A, Singi K, Kaulgud V. Personas and redundancies in crowdsourced testing. In: Proceedings of the 12th International Conference on Global Software Engineering; ICGSE '17. Buenos Aires, Argentinia: IEEE; 2017: 76-80
    Google Scholar
  • 29 Silva T, Silveira M, Maurer F. Usability evaluation practices within agile development. In: Proceedings of the 48th Hawaii International Conference on System Sciences. HICSS '15. Kauai, HI, USA: IEEE; 2015: 5133-5142
    Google Scholar
  • 30 Pawson M, Greenberg S. Extremely rapid usability testing. J Usability Stud 2009; 4: 124-135
    PubMedGoogle Scholar
  • 31 Kane D. Finding a place for discount usability engineering in agile development: throwing down the gauntlet. In: Proceedings of the Agile Development Conference; ADC '03. Salt Lake City, UT, USA: IEEE; 2003: 40-46
    Google Scholar
  • 32 Grigoreanu V, Mohanna M. Informal Cognitive Walkthroughs (ICW): paring down and pairing up for an agile world. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. Paris, France: ACM; 2013: 3093-3096
    Google Scholar
  • 33 Marques AB, Figueiredo R, Amorin W, Rabelo J, Barbosa SDJ, Conte T. Do usability and agility combine?: investigating the adoption of usability modeling in an agile software project in the industry. In: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems; IHC '18. Belém, Brazil: ACM; 2018: 1-11
    Google Scholar
  • 34 Cavichi de Freitas R, Rodrigues LA, Marques da Cunha A. AGILUS: a method for integrating usability evaluations on agile software development. In: Proceedings of the International Conference on Human-Computer Interaction; HCI '16. Cham, Switzerland: Springer International Publishing; 2016. ;9731: 545-552
    Google Scholar
  • 35 Magües D, Castro J, Acuna S. HCI usability techniques in agile development. In: Proceedings of the International Conference on Automatica; ICA-ACCA '16. Curico, Chile: IEEE; 2016: 1-7
    Google Scholar
  • 36 Butt S, Onn A, Butt M, Inam N, Butt S. Incorporation of usability evaluation methods in agile software model. In: Proceedings of the 17th IEEE International Multi Topic Conference; INMIC '14. Karachi, Pakistan: IEEE; 2014: 193-199
    Google Scholar
  • 37 Federoff M, Villamor C, Miller L. et al. Extreme usability: adapting research approaches for agile development. In: Proceedings of the 26th Annual CHI Conference Extended Abstracts on Human Factors in Computing Systems; CHI EA '08. Florence, Italy: ACM; 2008: 2269-2272
    Google Scholar
  • 38 Nielsen J. Discount usability: 20 years. Accessed September 13, 2019 at: https://www.nngroup.com/articles/discount-usability-20-years/
    Google Scholar
  • 39 Schwartz L. Agile-user experience design: an agile and user-centered process?. In: Proceedings of the 8th International Conference on Software Engineering Advances; ICSEA '13. Venice, Italy: IARIA; 2013: 346-351
    Google Scholar
  • 40 Isa W, Lokman A, Aris S. et al. Engineering rural informatics using agile user centered design. In: Proceedings of the 2nd International Conference on Information and Communication Technology; ICoICT '14. Bandung, Indonesia: IEEE; 2014: 367-372
    Google Scholar
  • 41 Constantine LL, Lockwood LAD. Usage-centered software engineering: an agile approach to integrating users, user interfaces, and usability into software engineering practice. In: Proceedings of the 25th International Conference on Software Engineering; ICSE '03. Portland, OR, USA: IEEE; 2003: 746-747
    Google Scholar
  • 42 Verhoeven F, van Gemert-Pijnen J. Discount user-centered e-health design: a quick-but-not-dirty method. In: Proceedings of the Symposium of the Austrian HCI and Usability Engineering Group; USAB '10. Klagenfurt, Austria: Springer; 2010. 638. 101-123
    Google Scholar
  • 43 Russ AL, Baker DA, Fahner WJ. et al. A rapid usability evaluation (RUE) method for health information technology. AMIA Annu Symp Proc 2010; 2010: 702-706
    PubMedGoogle Scholar
  • 44 European Union Agency for Network and Information Security (ENISA). Security and Resilience in eHealth: Security Challenges and Risks. Iraklio, Greece: ENISA; 2015: 1-48
    Google Scholar
  • 45 Kuziemsky CE, Kushniruk A. Context mediated usability testing. Stud Health Technol Inform 2014; 205: 905-909
    PubMedGoogle Scholar
  • 46 Øvad T, Larsen L. The prevalence of UX design in agile development processes in industry. In: Proceedings of the Agile Conference; AGILE '15. Washington, DC, USA: IEEE; 2015: 40-49
    Google Scholar
  • 47 Duh HB-L, Tan GCB, Chen VH. Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services; MobileHCI '06. Helsinki, Finland: ACM; 2006: 181-186
    Google Scholar
  • 48 Whitenton K. Tools for unmoderated usability testing. Accessed April 28, 2021 at: https://www.nngroup.com/articles/unmoderated-user-testing-tools/
    Google Scholar
  • 49 Nielsen J. Usability inspection methods. In: Proceedings of the Conference Companion on Human Factors in Computing Systems; CHI '94. Boston, MA, USA: ACM; 1994: 413-414
    Google Scholar
  • 50 Johnson CM, Johnston D, Crowley PK. et al. EHR Usability Toolkit: A Background Report on Usability and Electronic Health Records. Accessed August 8, 2021 at: https://digital.ahrq.gov/sites/default/files/docs/citation/EHR_Usability_Toolkit_Background_Report.pdf
    Google Scholar
  • 51 Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Curr HIV/AIDS Rep 2020; 17 (03) 203-218
    CrossrefPubMedGoogle Scholar
  • 52 Wu DTY, Vennemeyer S, Brown K. et al. Usability testing of an interactive dashboard for surgical quality improvement in a large congenital heart center. Appl Clin Inform 2019; 10 (05) 859-869
    Article in Thieme ConnectPubMedGoogle Scholar
  • 53 Elbabour F, Alhadreti O, Mayhew P. Eye tracking in retrospective think-aloud usability testing: is there added value?. J Usability Stud 2017; 12 (03) 95-110
    PubMedGoogle Scholar
  • 54 Elling S, Lentz L, de Jong M. Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations. In: Proceedings of the 29th Annual CHI Conference on Human Factors in Computing Systems; SIGCHI '11. Vancouver, BC, Canada: ACM; 2011: 1161-1170
    Google Scholar
  • 55 Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc 2017; 24 (e1): e55-e60
    CrossrefPubMedGoogle Scholar
  • 56 Georgsson M, Staggers N, Årsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: a case study and beginning method validation. J Biomed Inform 2019; 91: 1-15
    CrossrefPubMedGoogle Scholar
  • 57 Ghalibaf AK, Jangi M, Habibi MRM, Zangouei S, Khajouei R. Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electron Physician 2018; 10 (04) 6682-6688
    CrossrefPubMedGoogle Scholar
  • 58 Rowley DE, Rhoades DG. The cognitive jogthrough: a fast-paced user interface evaluation procedure. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; CHI '92. Monterey, Calif, USA: ACM; 1992: 389-395
    Google Scholar
  • 59 Andersen SB, Rasmussen CK, Frøkjær E. Bringing content understanding into usability testing in complex application domains - a case study in eHealth. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 327-341
    Google Scholar
  • 60 The User Experience Professionals' Association (UXPA). Usability body of knowledge. Accessed August 9, 2021 at: http://www.usabilitybok.org/methods
    Google Scholar
  • 61 University of Minnesota Duluth (UMD). Usability evaluation toolbox. Accessed August 15, 2021 at: https://www.d.umn.edu/itss/training/online/usability/toolbox.html
    Google Scholar
  • 62 Maguire M. Methods to support human-centred design. Int J Hum Comput Stud 2001; 55 (04) 587-634
    CrossrefPubMedGoogle Scholar
  • 63 Frøkjær E, Hornbæk K. Cooperative usability testing. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '05. Portland, OR, USA: ACM; 2005: 1383-1386
    Google Scholar
  • 64 Power C, Petrie H, Mitchell R. A framework for remote user evaluation of accessibility and usability of websites. In: Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction; UAHCI '09. San Diego, CA, USA: Springer; 2009. ;5614: 594-601
    Google Scholar
  • 65 Hammontree M, Weiler P, Nayak N. Remote usability testing. Interact 1994; 1 (03) 21-25
    CrossrefPubMedGoogle Scholar
  • 66 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
    CrossrefPubMedGoogle Scholar
  • 67 Medlock MC, Wixon D, McGee M, Welsh D. The rapid iterative test and evaluation method. In: Int Techn, Cost-Justifying Usability. 2nd ed.. Elsevier; 2005: 489-517
    Google Scholar
  • 68 Wilson C. User Interface Inspection Methods: A User-Centered Design Method. Amsterdam, The Netherlands; Elsevier. 2013
    Google Scholar
  • 69 Laemller R. Guerrilla Testing: Muting und rasch zu Resultaten. Accessed November 2, 2020 at: https://www.testingtime.com/blog/guerrilla-testing/
    Google Scholar
  • 70 Firmenich S, Garrido A, Grigera J, Rivero JM, Rossi G. Usability improvement through A/B testing and refactoring. Softw Qual J 2018; 27 (01) 203-240
    CrossrefPubMedGoogle Scholar
  • 71 Guaiani F, Muccini H. Crowd and laboratory testing, can they co-exist? An exploratory study. In: Proceedings of the 2nd International Workshop on CrowdSourcing in Software Engineering; CSI-SE '15. Florence, Italy: IEEE; 2015: 32-37
    Google Scholar
  • 72 Zogaj S, Bretschneider U, Leimeister JM. Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary. J Bus Econ 2014; 84 (03) 375-405
    CrossrefPubMedGoogle Scholar
  • 73 Downey LL. Group usability testing: evolution in usability techniques. J Usability Stud 2007; 2 (03) 133-144
    PubMedGoogle Scholar
  • 74 Sears A. Heuristic walkthroughs: finding the problems without the noise. Int J Hum Comput Interact 1997; 9 (03) 213-234
    CrossrefPubMedGoogle Scholar
  • 75 Nielsen L, Madsen S. The usability expert's fear of agility: an empirical study of global trends and emerging practices. In: Proceedings of the 7th Nordic Conference on Human-Computer Interaction; NordiCHI '12. Copenhagen, Denmark: ACM; 2012: 261-264
    Google Scholar
  • 76 Höysniemi J, Hämäläinen P, Turkki L. Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interact Comput 2003; 15 (02) 203-255
    CrossrefPubMedGoogle Scholar
  • 77 Khalayli N, Nyhus S, Hamnes K, Terum T. Persona based rapid usability kick-off. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '07. San Jose, CA, USA: ACM; 2007: 1771-1776
    Google Scholar
  • 78 Usability in Germany eV (UIG). Fokusgruppe. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fokusgruppe
    Google Scholar
  • 79 Rosenbaum S, Cockton G, Coyne K, Muller M, Rauch T. Focus groups in HCI. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '02. Minneapolis, MN, USA: ACM; 2002: 702-703
    Google Scholar
  • 80 Usability in Germany eV (UIG). Fragebogen. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fragebogen
    Google Scholar
  • 81 Krause R. Storyboards help visualize UX ideas. Nielsen Norman Group. Accessed November 9, 2020 at: https://www.nngroup.com/articles/storyboards-visualize-ideas/
    Google Scholar
  • 82 Piyush J, Sanjay KD, Ajay R. Software usability evaluation method. Int J Adv Res Comput Eng Technol 2012; 1 (02) 28-33
    PubMedGoogle Scholar
  • 83 Holzinger A, Slany WXP. + UE → XU Praktische Erfahrungen mit eXtreme Usability. Informatik-Spektrum 2006; 29 (02) 91-97
    CrossrefPubMedGoogle Scholar
  • 84 Borycki E, Senathirajah Y, Kushniruk AW. The future of mobile usability, workflow and safety testing. Stud Health Technol Inform 2017; 245: 15-19
    PubMedGoogle Scholar
  • 85 Holzinger A. Usability engineering methods for software developers. Commun ACM 2005; 48 (01) 71-74
    CrossrefPubMedGoogle Scholar