Evaluating the Efficiency and Safety of Speech Recognition within a Commercial Electronic Health Record System: A Replication Study

Abstract

Objective To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system.

Methods Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed.

Results Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9–48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5–2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5–1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0–0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0–3.0).

Discussion The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results.

Conclusion This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

Keywords

electronic health record - speech recognition - integration - medical errors - patient safety

Authors' Contributions

T.H., E.C., and F.M. conceived the study and its design. T.H. conducted the research, the primary analysis, and the initial drafting of the manuscript. E.C. and F.M. contributed to the analysis and drafting of the manuscript, and T.H., E.C., and F.M. approved the final manuscript. T.H. is the corresponding author.

Protection of Human and Animal Subjects

This study is approved by the Sydney Local Health District Local Health District Human Ethics Committee—Concord Repatriation and General Hospital (CRGH) (LNR/14/CRGH/272).

• References

• 1 Hodgson T, Coiera E. Risks and benefits of speech recognition for clinical documentation: a systematic review. J Am Med Inform Assoc 2016; 23 (e1): e169-e179
• 2 Hodgson T, Magrabi F, Coiera E. Efficiency and safety of speech recognition for documentation in the electronic health record. J Am Med Inform Assoc 2017; 24 (06) 1127-1133
• 3 Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 2015; 349 (6251): aac4716
• 4 Freedman LP, Cockburn IM, Simcoe TS. The economics of reproducibility in preclinical research. PLoS Biol 2015; 13 (06) e1002165
• 5 Nakagawa S, Parker TH. Replicating research in ecology and evolution: feasibility, incentives, and the cost-benefit conundrum. BMC Biol 2015; 13 (01) 88
• 6 Coiera E, Ammenwerth E, Georgiou A, Magrabi F. Does health informatics have a replication crisis?. J Am Med Inform Assoc 2018; DOI: 10.1093/jamia/ocy028.
• 7 Engeström Y. Activity theory as a framework for analyzing and redesigning work. Ergonomics 2000; 43 (07) 960-974
• 8 Kaptelinin V, Nardi BA, Macaulay C. Methods & tools: the activity checklist: a tool for representing the “space” of context. Interactions 1999; 6 (04) 27-39
• 9 Guidance for Industry and Food and Drug Administration Staff. Guidance for the content of premarket submissions for software contained in medical devices. Rockville, MD: Center for Devices and Radiological Health; 2005
• 10 Castaneda C, Nalley K, Mannion C. , et al. Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. J Clin Bioinforma 2015; 5 (01) 4
• 11 Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support?. J Biomed Inform 2009; 42 (05) 760-772
• 12 Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011; 18 (03) 327-334
• 13 Gross MT. The Need for Replication Studies—Is It Really a Done Deal?. J Orthop Sports Phys Ther 1997; 25 (03) 161-162