Download PDF
Appl Clin Inform 2024; 15(05): 1039-1048
DOI: 10.1055/a-2404-2129
State of the Art/Best Practice Paper

Realizing the Full Potential of Clinical Decision Support: Translating Usability Testing into Routine Practice in Health Care Operations

Swaminathan Kandaswamy
1   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
,
Herbert Williams
2   Division of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
Sarah Anne Thompson
2   Division of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
Thomas Elijah Dawson
2   Division of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
Naveen Muthu
1   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
2   Division of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
3   Division of Hospital Medicine, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
Evan William Orenstein
1   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
2   Division of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
3   Division of Hospital Medicine, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
› Author Affiliations
› Further Information
Also available at
    Permissions and Reprints

Abstract

Background Clinical Decision Support (CDS) tools have a mixed record of effectiveness, often due to inadequate alignment with clinical workflows and poor usability. While there is a consensus that usability testing methods address these issues, in practice, usability testing is generally only used for selected projects (such as funded research studies). There is a critical need for CDS operations to apply usability testing to all CDS implementations.

Objectives In this State of the Art/Best Practice paper, we share challenges with scaling usability in health care operations and alternative methods and CDS governance structures to enable usability testing as a routine practice.

Methods We coalesce our experience and results of applying guerilla in situ usability testing to over 20 projects in a 1-year period with the proposed solution.

Results We demonstrate the feasibility of adopting “guerilla in situ usability testing” in operations and their effectiveness in incorporating user feedback and improving design.

Conclusion Although some methodological rigor was relaxed to accommodate operational speed, the benefits outweighed the limitations. Broader adoption of usability testing may transform CDS implementation and improve health outcomes.

Keywords

clinical decision support - usability testing - translating research

Protection of Human Subjects

No human subjects were involved in this perspective. In discussion with the Children's Healthcare of Atlanta IRB, projects applying guerilla in situ usability testing were deemed as quality improvement projects and therefore nonhuman subjects research.




Publication History

Received: 03 June 2024

Accepted: 25 August 2024

Accepted Manuscript online:
27 August 2024

Article published online:
04 December 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Shultz S, Hand MW. Usability: a concept analysis. . J Theory Constr Test 2015;19(02):
    PubMed
  • 2 Department of Health and Human Services. Usability Evaluation Basics. 2013 . Accessed September 29, 2022 at: https://www.usability.gov/what-and-why/usability-evaluation.html
    PubMedSearch in Google Scholar
  • 3 Nielsen Norman Group. Usability 101: Introduction to Usability. Accessed June 8, 2023 at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
    PubMed
  • 4 Pruitt Z, Howe JL, Krevat SA, Khairat S, Ratwani RM. Development and pilot evaluation of an electronic health record usability and safety self-assessment tool. JAMIA Open 2022; 5 (03) ooac070
    CrossrefPubMedSearch in Google Scholar
  • 5 Park S, Marquard J, Austin R, Pieczkiewicz D, Delaney C. Usability of electronic health records from nurses' perspectives: a systematic review. Paper presented at: 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI). 2022: 511-512
    PubMedSearch in Google Scholar
  • 6 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
    CrossrefPubMedSearch in Google Scholar
  • 7 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
    CrossrefPubMedSearch in Google Scholar
  • 8 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
    CrossrefPubMedSearch in Google Scholar
  • 9 Khairat S, Burke G, Archambault H, Schwartz T, Larson J, Ratwani RM. Perceived burden of EHRs on physicians at different stages of their career. Appl Clin Inform 2018; 9 (02) 336-347
    Thieme ConnectPubMedSearch in Google Scholar
  • 10 Gomes KM, Ratwani RM. Evaluating improvements and shortcomings in clinician satisfaction with electronic health record usability. JAMA Netw Open 2019; 2 (12) e1916651
    CrossrefPubMedSearch in Google Scholar
  • 11 Koppel R, Metlay JP, Cohen A. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203
    CrossrefPubMedSearch in Google Scholar
  • 12 Ratwani R, Fairbanks T, Savage E. et al. Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation. Appl Clin Inform 2016; 7 (04) 1069-1087
    Thieme ConnectPubMedSearch in Google Scholar
  • 13 Dixit RA, Boxley CL, Samuel S, Mohan V, Ratwani RM, Gold JA. Electronic health record use issues and diagnostic error: a scoping review and framework. J Patient Saf 2023; 19 (01) e25-e30
    CrossrefPubMedSearch in Google Scholar
  • 14 Kariotis TC, Prictor M, Chang S, Gray K. Impact of electronic health records on information practices in mental health contexts: scoping review. J Med Internet Res 2022; 24 (05) e30405
    CrossrefPubMedSearch in Google Scholar
  • 15 Li RC, Wang JK, Sharp C, Chen JH. When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record. BMJ Qual Saf 2019; 28 (12) 987-996
    PubMedSearch in Google Scholar
  • 16 Reese TJ, Liu S, Steitz B. et al. Conceptualizing clinical decision support as complex interventions: a meta-analysis of comparative effectiveness trials. J Am Med Inform Assoc 2022; 29 (10) 1744-1756
    CrossrefPubMedSearch in Google Scholar
  • 17 Orenstein EW, Weitkamp AO, Rosenau P, Mallozzi C, Tobias MC. Towards a maturity model for clinical decision support operations: an interactive panel. Paper presented at: American Medical Informatics Association Clinical Informatics Conference. Atlanta, GA; 2019
    PubMedSearch in Google Scholar
  • 18 Kushniruk A, Borycki E, Kuo MH, Kuwata S. Integrating technology-centric and user-centric system testing methods: ensuring healthcare system usability and safety. In: Information Technology in Health Care: Socio-Technical Approaches 2010. IOS Press; 2010: 181-186 . Accessed November 8, 2023 at: https://ebooks.iospress.nl/doi/10.3233/978-1-60750-569-3-181
    Search in Google Scholar
  • 19 Desai AV, Michael CL, Kuperman GJ. et al. A novel patient values tab for the electronic health record: a user-centered design approach. J Med Internet Res 2021; 23 (02) e21615
    CrossrefPubMedSearch in Google Scholar
  • 20 Blanes-Selva V, Asensio-Cuesta S, Doñate-Martínez A, Pereira Mesquita F, García-Gómez JM. User-centred design of a clinical decision support system for palliative care: Insights from healthcare professionals. Digit Health 2023; 9: 20 552076221150735
    PubMedSearch in Google Scholar
  • 21 Rudin RS, Perez S, Rodriguez JA. et al. User-centered design of a scalable, electronic health record-integrated remote symptom monitoring intervention for patients with asthma and providers in primary care. J Am Med Inform Assoc 2021; 28 (11) 2433-2444
    CrossrefPubMedSearch in Google Scholar
  • 22 Horsky J, Ramelson HZ. Development of a cognitive framework of patient record summary review in the formative phase of user-centered design. J Biomed Inform 2016; 64: 147-157
    CrossrefPubMedSearch in Google Scholar
  • 23 Chokshi SK, Belli HM, Troxel AB. et al. Designing for implementation: user-centered development and pilot testing of a behavioral economic-inspired electronic health record clinical decision support module. Pilot Feasibility Stud 2019; 5 (01) 28
    CrossrefPubMedSearch in Google Scholar
  • 24 Thursky KA, Mahemoff M. User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit. Int J Med Inform 2007; 76 (10) 760-768
    CrossrefPubMedSearch in Google Scholar
  • 25 Nguyen KA, Patel H, Haggstrom DA, Zillich AJ, Imperiale TF, Russ AL. Utilizing a user-centered approach to develop and assess pharmacogenomic clinical decision support for thiopurine methyltransferase. BMC Med Inform Decis Mak 2019; 19 (01) 194
    CrossrefPubMedSearch in Google Scholar
  • 26 Miller SD, Murphy Z, Gray JH. et al. Human-centered design of a clinical decision support for anemia screening in children with inflammatory bowel disease. Appl Clin Inform 2023; 14 (02) 345-353
    Thieme ConnectPubMedSearch in Google Scholar
  • 27 Molloy MJ, Zackoff M, Gifford A. et al. Usability testing of situation awareness clinical decision support in the intensive care unit. Appl Clin Inform 2024; 15 (02) 327-334
    Thieme ConnectPubMedSearch in Google Scholar
  • 28 Shear K, Rice H, Garabedian PM. et al. Usability testing of an interoperable computerized clinical decision support tool for fall risk management in primary care. Appl Clin Inform 2023; 14 (02) 212-226
    Thieme ConnectPubMedSearch in Google Scholar
  • 29 McGonagle EA, Karavite DJ, Grundmeier RW. et al. Evaluation of an antimicrobial stewardship decision support for pediatric infections. Appl Clin Inform 2023; 14 (01) 108-118
    Thieme ConnectPubMedSearch in Google Scholar
  • 30 McNab D, Freestone J, Black C, Carson-Stevens A, Bowie P. Participatory design of an improvement intervention for the primary care management of possible sepsis using the Functional Resonance Analysis Method. BMC Med 2018; 16 (01) 174
    CrossrefPubMedSearch in Google Scholar
  • 31 Beerlage-de Jong N, Wentzel J, Hendrix R, van Gemert-Pijnen L. The value of participatory development to support antimicrobial stewardship with a clinical decision support system. Am J Infect Control 2017; 45 (04) 365-371
    CrossrefPubMedSearch in Google Scholar
  • 32 van Leeuwen D, Mittelman M, Fabian L, Lomotan EA. Nothing for me or about me, without me: codesign of clinical decision support. Appl Clin Inform 2022; 13 (03) 641-646
    Thieme ConnectPubMedSearch in Google Scholar
  • 33 Kushniruk A, Nohr C. Participatory design, user involvement and health IT evaluation. In: Evidence-Based Health Informatics. IOS Press; 2016. :139–151. Accessed July 30, 2024 at: https://ebooks.iospress.nl/doi/10.3233/978-1-61499-635-4-139
    Search in Google Scholar
  • 34 Perry SJ, Catchpole K, Rivera AJ, Henrickson Parker S, Gosbee J. 'Strangers in a strange land': Understanding professional challenges for human factors/ergonomics and healthcare. Appl Ergon 2021; 94: 103040
    CrossrefPubMedSearch in Google Scholar
  • 35 Nielsen J, Levy J. Measuring usability: preference vs. performance. Commun ACM 1994; 37 (04) 66-75
    CrossrefPubMedSearch in Google Scholar
  • 36 Staggers N, Rodney M. Promoting Usability in Organizations with a New Health Usability Model: Implications for Nursing Informatics. NI 2012: 11th International Congress on Nursing Informatics; June 23–27, 2012; Montreal, Canada. 2012: 396 . Accessed June 8, 2023 at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3799150/
    PubMedSearch in Google Scholar
  • 37 Shah T, Kitts AB, Gold JA. et al. Electronic health record optimization and clinician well-being: a potential roadmap toward action. NAM Perspect 2020; 2020
    CrossrefPubMedSearch in Google Scholar
  • 38 Mann DM, Chokshi SK, Kushniruk A. Bridging the gap between academic research and pragmatic needs in usability: a hybrid approach to usability evaluation of health care information systems. JMIR Hum Factors 2018; 5 (04) e10721
    CrossrefPubMedSearch in Google Scholar
  • 39 Kushniruk AW, Borycki EM, Kannry J. Commercial versus in-situ usability testing of healthcare information systems: towards “public” usability testing in healthcare organizations. Stud Health Technol Inform 2013; 183: 157-161
    PubMedSearch in Google Scholar
  • 40 Nielsen Norman Group. Discount Usability: 20 Years. Accessed May 14, 2024 at: https://www.nngroup.com/articles/discount-usability-20-years/
    PubMed
  • 41 Motulsky SL. Is member checking the gold standard of quality in qualitative research?. Qual Psychol 2021; 8 (03) 389-406
    CrossrefPubMedSearch in Google Scholar
  • 42 Holden RJ, Carayon P. SEIPS 101 and seven simple SEIPS tools. BMJ Qual Saf 2021; 30 (11) 901-910
    CrossrefPubMedSearch in Google Scholar
  • 43 McDonald S, Edwards HM, Zhao T. Exploring Think-Alouds in Usability Testing: an international survey. IEEE Trans Prof Commun 2012; 55 (01) 2-19
    CrossrefPubMedSearch in Google Scholar
  • 44 EDawson T, Beus J, WOrenstein E, Umontuen U, McNeill D, Kandaswamy S. Reducing therapeutic duplication in inpatient medication orders. Appl Clin Inform 2023; 14 (03) 538-543
    Thieme ConnectPubMedSearch in Google Scholar
  • 45 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
    Thieme ConnectPubMedSearch in Google Scholar
  • 46 Orenstein EW, Rollins M, Jones J. et al. Influence of user-centered clinical decision support on pediatric blood product ordering errors. Blood Transfus Trasfus Sangue 2023; 21: 3-12
    PubMedSearch in Google Scholar
  • 47 Thompson SA, Williams H, Rzewnicki D. et al. Avoiding Unintended Consequences of Pediatric Blood Order Set Updates through In Situ Usability Testing. Appl Clin Inform. Accessed July 23, 2024 at: https://www.thieme-connect.com/products/ejournals/abstract/10.1055/a-2351-9642
    PubMed
  • 48 Kandaswamy S, Jones J, Orenstein EW. User Center Design of Blood Orders in the EHR to reduce risk of over-transfusion in a pediatric healthcare system. Paper presented at : AABB Annual Meeting. Virtual Meeting: Wiley Online Library; 2020
    PubMedSearch in Google Scholar
  • 49 Orenstein EW, Rasooly IR, Mai MV. et al. Influence of simulation on electronic health record use patterns among pediatric residents. J Am Med Inform Assoc 2018; 25 (11) 1501-1506
    CrossrefPubMedSearch in Google Scholar
  • 50 Orenstein EW, ElSayed-Ali O, Kandaswamy S. et al. Evaluation of a clinical decision support strategy to increase seasonal influenza vaccination among hospitalized children before inpatient discharge. JAMA Netw Open 2021; 4 (07) e2117809-e2117809
    CrossrefPubMedSearch in Google Scholar
  • 51 Mrosak J, Kandaswamy S, Stokes C. et al. The influence of integrating clinical practice guideline order bundles into a general admission order set on guideline adoption. JAMIA Open 2021; 4 (04) ooab087
    CrossrefPubMedSearch in Google Scholar
  • 52 Mrosak J, Kandaswamy S, Stokes C, Roth D, Orenstein E. Formative usability testing of an admission order set with modular disease-specific order groups to improve guideline order set usage. Paper presented at: Pediatric Hospital Medicine Conference. 2020. ; Florida
    Search in Google Scholar
  • 53 Kandaswamy S, Gill A, Wood S. et al. User-centered design of central venous access device documentation. JAMIA Open 2022; 5 (01) ooac011
    CrossrefPubMedSearch in Google Scholar
  • 54 Kandaswamy S, Jain S, Dwight C. et al. Improving outcomes for pediatric patients with metabolic conditions in the ED using user centered design for order sets. Paper presented at: AMIA 2021 Annual Symposium. November 30, 2021 ; San Diego.
    PubMedSearch in Google Scholar
  • 55 Bennett B, Kurzen E, Orenstein E, Kandaswamy S, Shin H. Clinical decision support to reduce nephrotoxic medication-associated acute kidney injury in non-critically ill hospitalized children. Paper presented at: AMIA Annual Meeting 2022. November 5, 2022 ; Washington, D.C.
    PubMedSearch in Google Scholar
  • 56 Rowland AF, Nguyen TH, Cunha PP. et al. Implementing a clinical decision support tool to increase early peanut introduction guidance. J Allergy Clin Immunol 2024; S0091 -6749(24)00712-7
    PubMedSearch in Google Scholar
  • 57 Thompson SA, Kandaswamy S, Orenstein E. CIC 2023: A discount approach to reducing nursing alert burden. Appl Clin Inform 2024; 15: 727-732
    Thieme ConnectPubMedSearch in Google Scholar
  • 58 Siegel BI, Johnson M, Dawson TE. et al. Reducing prescribing errors in hospitalized children on the ketogenic diet. Pediatr Neurol 2021; 115: 42-47
    CrossrefPubMedSearch in Google Scholar
  • 59 Mrosak J, Kandaswamy S, Stokes C. et al. The effect of implementation of guideline order bundles into a general admission order set on clinical practice guideline adoption: quasi-experimental study. JMIR Med Inform 2023; 11: e42736
    CrossrefPubMedSearch in Google Scholar
  • 60 Kandaswamy S, Masterson E, Blanco R. et al. Barriers to seasonal influenza vaccine uptake in a pediatric inpatient healthcare setting after implementation of clinical decision support. Stud Health Technol Inform 2022; 290: 452-456
    PubMedSearch in Google Scholar
  • 61 Siegel B, Shahnawaz M, Elkins K, Kheder A, Orenstein E. Validation of a semi-automated method for tracking quality metrics in benzodiazepine-resistant status epilepticus (S7.007). . Neurology 2022;98(18 Supplement). Accessed June 13, 2023 at: https://n.neurology.org/content/98/18_Supplement/706
    PubMed
  • 62 Thompson S, Mckay L, Ray E, Kandaswamy S, Orenstein EW. Improve or remove: challenges in refining ineffective alerts – the case of elopement prevention. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
    PubMedSearch in Google Scholar
  • 63 Nguyen TH, Cunha PP, Rowland AF, Orenstein E, Lee T, Kandaswamy S. User-centered design and evaluation of clinical decision support to improve early peanut introduction: formative study. JMIR Form Res 2023; 7 (01) e47574
    CrossrefPubMedSearch in Google Scholar
  • 64 Cunha P, Nguyen T, Pham T, Kandaswamy S, Lee T. Implementing early peanut introduction recommendations by pediatric residents through a clinical decision support system. J Allergy Clin Immunol 2023; 151 (02) AB44 . Accessed March 10, 2023 at: https://www.jacionline.org/article/S0091-6749(22)01792-4/fulltext
    CrossrefPubMedSearch in Google Scholar
  • 65 Dawson TE, Beus JM, Orenstein E, Kurzen E, Kandaswamy S. Decision support to improve safety of discharge prescriptions of subcutaneous medications. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
    PubMedSearch in Google Scholar
  • 66 Kandaswamy S, Thompson S, Orenstein E. Accurate dosing weight: when the 10% really matters. Stud Health Technol Inform 2024; 310: 354-358
    PubMedSearch in Google Scholar
  • 67 Kandaswamy S, Dawson TE, Shin H, Orenstein EW. Non-interruptive alerts for reduced renal function: does anyone see them?. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
    PubMedSearch in Google Scholar
  • 68 Thompson S, Orenstein EW, Kandaswamy S. Accurate dosing weight: when the 10% really matters. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
    PubMedSearch in Google Scholar
  • 69 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77-101 Accessed May 14, 2024 at: https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa
    CrossrefPubMedSearch in Google Scholar
  • 70 Stead WW, Haynes RB, Fuller S. et al. Designing medical informatics research and library–resource projects to increase what is learned. J Am Med Inform Assoc 1994; 1 (01) 28-33
    CrossrefPubMedSearch in Google Scholar
  • 71 Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002; 32 (03) 141-149
    CrossrefPubMedSearch in Google Scholar
  • 72 Kaufman D, Roberts WD, Merrill J, Lai TY, Bakken S. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res 2006; 55 (2 Suppl): S37-S42
    CrossrefPubMedSearch in Google Scholar
  • 73 Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems-problems and challenges. Int J Med Inform 2003; 71 (2–3): 125-135
    CrossrefPubMedSearch in Google Scholar