Appl Clin Inform 2023; 14(02): 212-226
DOI: 10.1055/a-2006-4936
Research Article

Usability Testing of an Interoperable Computerized Clinical Decision Support Tool for Fall Risk Management in Primary Care

Kristen Shear
1   Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, Florida, United States
Hannah Rice
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
Pamela M. Garabedian
3   Department of Information Systems, Mass General Brigham, Somerville, Massachusetts, United States
Ragnhildur Bjarnadottir
1   Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, Florida, United States
Nancy Lathum
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
Ann L. Horgas
4   Department of Biobehavioral Nursing Science, College of Nursing, University of Florida, Gainesville, Florida, United States
Christopher A. Harle
5   Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, United States
Patricia C. Dykes
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
Robert Lucero
6   UCLA School of Nursing, University of California Los Angeles, Los Angeles, California, United States
› Author Affiliations
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this work was supported by the Agency for Healthcare Research and Quality [grant number: U18HS027557].


Background Falls are a widespread and persistent problem for community-dwelling older adults. Use of fall prevention guidelines in the primary care setting has been suboptimal. Interoperable computerized clinical decision support systems have the potential to increase engagement with fall risk management at scale. To support fall risk management across organizations, our team developed the ASPIRE tool for use in differing primary care clinics using interoperable standards.

Objectives Usability testing of ASPIRE was conducted to measure ease of access, overall usability, learnability, and acceptability prior to pilot .

Methods Participants were recruited using purposive sampling from two sites with different electronic health records and different clinical organizations. Formative testing rooted in user-centered design was followed by summative testing using a simulation approach. During summative testing participants used ASPIRE across two clinical scenarios and were randomized to determine which scenario they saw first. Single Ease Question and System Usability Scale were used in addition to analysis of recorded sessions in NVivo.

Results All 14 participants rated the usability of ASPIRE as above average based on usability benchmarks for the System Usability Scale metric. Time on task decreased significantly between the first and second scenarios indicating good learnability. However, acceptability data were more mixed with some recommendations being consistently accepted while others were adopted less frequently.

Conclusion This study described the usability testing of the ASPIRE system within two different organizations using different electronic health records. Overall, the system was rated well, and further pilot testing should be done to validate that these positive results translate into clinical practice. Due to its interoperable design, ASPIRE could be integrated into diverse organizations allowing a tailored implementation without the need to build a new system for each organization. This distinction makes ASPIRE well positioned to impact the challenge of falls at scale.

Protection of Human and Animal Subjects

IRB approval was received for protocol number: Site 1: 2020P002075; Site 2: CED000000426_.


P.C.D. and R.L. contributed equally to the development of the manuscript as co–senior investigators.

The views expressed herein are those of the author(s) and do not reflect the official policy or position of Brooke Army Medical Center, the Department of Defense, or any agencies under the U.S. Government.

Publication History

Received: 07 July 2022

Accepted: 02 January 2023

Accepted Manuscript online:
04 January 2023

Article published online:
15 March 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Florence CS, Bergen G, Atherly A, Burns E, Stevens J, Drake C. Medical costs of fatal and nonfatal falls in older adults. J Am Geriatr Soc 2018; 66 (04) 693-698
  • 2 Moreland B, Kakara R, Henry A. Trends in nonfatal falls and fall-related injuries among adults aged ≥65 years - United States, 2012-2018. MMWR Morb Mortal Wkly Rep 2020; 69 (27) 875-881
  • 3 Clinical Decision Support. The Office of the National Coordinator for Health Information Technology. Accessed March 7, 2021 at:
  • 4 Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care 2008; 16 (02) 79-91
  • 5 Trinkley KE, Kroehl ME, Kahn MG. et al. Applying clinical decision support design best practices with the practical robust implementation and sustainability model versus reliance on commercially available clinical decision support tools: randomized controlled trial. JMIR Med Inform 2021; 9 (03) e24359
  • 6 Taylor SF, Coogle CL, Cotter JJ, Welleford EA, Copolillo A. Community-dwelling older adults' adherence to environmental fall prevention recommendations. J Appl Gerontol 2019; 38 (06) 755-774
  • 7 Assurance NCfQ. Fall risk management. National Committee for Quality Assurance. Accessed April 25, 2022 at:
  • 8 Howland J, Hackman H, Taylor A, O'Hara K, Liu J, Brusch J. Older adult fall prevention practices among primary care providers at accountable care organizations: a pilot study. PLoS One 2018; 13 (10) e0205279
  • 9 Spears GV, Roth CP, Miake-Lye IM, Saliba D, Shekelle PG, Ganz DA. Redesign of an electronic clinical reminder to prevent falls in older adults. Med Care 2013; 51 (03) (3, suppl 1): S37-S43
  • 10 Zheng MY, Suneja A, Chou AL, Arya M. Physician barriers to successful implementation of US Preventive Services Task Force routine HIV testing recommendations. J Int Assoc Provid AIDS Care 2014; 13 (03) 200-205
  • 11 Hoskins KF, Tejeda S, Vijayasiri G. et al. A feasibility study of breast cancer genetic risk assessment in a federally qualified health center. Cancer 2018; 124 (18) 3733-3741
  • 12 Kurth AE, Krist AH, Borsky AE. et al. U.S. Preventive Services Task Force methods to communicate and disseminate clinical preventive services recommendations. Am J Prev Med 2018; 54 (1S1): S81-S87
  • 13 Braunstein ML. Health Informatics on FHIR: How HL7's New API is Transforming Healthcare. Cham: Springer; 2018: 292
  • 14 Barnett ML, Bitton A, Souza J, Landon BE. Trends in outpatient care for medicare beneficiaries and implications for primary care, 2000 to 2019. Ann Intern Med 2021; 174 (12) 1658-1665
  • 15 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
  • 16 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
  • 17 Nielsen J. Chapter 2 - What is usability? In: Nielsen J, ed. Usability Engineering. Burlington, MA: Morgan Kaufmann; 1993: 23-48
  • 18 Peute LW, Spithoven R, Bakker PJ, Jaspers MW. Usability studies on interactive health information systems; where do we stand?. Stud Health Technol Inform 2008; 136: 327-332
  • 19 Nielsen J, Mack RL. Usability Inspection Methods. Morristown, NJ: Wiley; 1994
  • 20 Li AC, Kannry JL, Kushniruk A. et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012; 81 (11) 761-772
  • 21 Rice H, Garabedian PM, Shear K. et al. Clinical decision support for fall prevention: defining end-user needs. Appl Clin Inform 2022; 13 (03) 647-655
  • 22 Rice HG, Garabedian P, Shear K. et al. Computerized clinical decision support for fall prevention: defining end-user requirements for primary care staff and patients. Paper presented at: 2022 AMIA Informatics Summit; March 21–24, 2022; Chicago, IL, United States. Accessed January 18, 2023 at:
  • 23 Grossman DC, Curry SJ, Owens DK. et al; US Preventive Services Task Force. Interventions to prevent falls in community-dwelling older adults: US Preventive Services Task Force recommendation statement. JAMA 2018; 319 (16) 1696-1704
  • 24 Shubert TE, Smith ML, Jiang L, Ory MG. Disseminating the Otago Exercise Program in the United States: perceived and actual physical performance improvements from participants. J Appl Gerontol 2018; 37 (01) 79-98
  • 25 De Vito Dabbs A, Myers BA, Mc Curry KR. et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009; 27 (03) 175-183
  • 26 Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. 2nd ed. Amsterdam: Elsevier; 2016: 350
  • 27 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
  • 28 Albert W, Tullis TS. Chapter 5 - Self-reported metrics. In: Albert W, Tullis TS, eds. Measuring the User Experience. 3rd ed. Burlington, MA: Morgan Kaufmann; 2023: 109-151
  • 29 Lewis JR, Sauro J. The Factor Structure of the System Usability Scale. Berlin, Heidelberg: Springer; 2009: 94-103
  • 30 Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F. Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Hum Comput Interact 2015; 31 (08) 484-495
  • 31 Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017; 17 (01) 88
  • 32 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
  • 33 Connect Repository CDS. Agency for Healthcare Research and Quality. Accessed April 29, 2022 at:
  • 34 Camacho PM, Petak SM, Binkley N. et al. American Association of Clinical Endocrinologists/American College of Endocrinology Clinical Practice Guidelines for the Diagnosis and Treatment of Postmenopausal Osteoporosis-2020 Update. Endocr Pract 2020; 26 (suppl 1): 1-46
  • 35 Weinfeld JM, Gorman PN. Primary care physician designation and response to clinical decision support reminders: a cross-sectional study. Appl Clin Inform 2016; 7 (02) 248-259