Subscribe to RSS
Accuracy of Physician Electronic Health Record Usage Analytics using Clinical Test CasesFunding There was no formal source of funding for this work. In-kind support was provided by the Centre for Addiction and Mental Health.
Usage log data are an important data source for characterizing the potential burden related to use of the electronic health record (EHR) system. However, the utility of this data source has been hindered by concerns related to the real-world validity and accuracy of the data. While time–motion studies have historically been used to address this concern, the restrictions caused by the pandemic have made it difficult to carry out these studies in-person. In this regard, we introduce a practical approach for conducting validation studies for usage log data in a controlled environment. By developing test runs based on clinical workflows and conducting them within a test EHR environment, it allows for both comparison of the recorded timings and retrospective investigation of any discrepancies. In this case report, we describe the utility of this approach for validating our physician EHR usage logs at a large academic teaching mental health hospital in Canada. A total of 10 test runs were conducted across 3 days to validate 8 EHR usage log metrics, finding differences between recorded measurements and the usage analytics platform ranging from 9 to 60%.
Data Availability Statement
The validation data from this project can be shared upon reasonable request to the corresponding author.
Protection of Human and Animal Subjects
The study was reviewed by CAMH Quality Project Ethics Review Board.
* These authors contributed equally.
Received: 06 May 2022
Accepted: 25 July 2022
Article published online:
05 October 2022
© 2022. Thieme. All rights reserved.
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
- 1 Moy AJ, Schwartz JM, Chen R. et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. J Am Med Inform Assoc 2021; 28 (05) 998-1008
- 2 Kannampallil TG, Denton CA, Shapiro JS, Patel VL. Efficiency of emergency physicians: insights from an observational study using EHR log files. Appl Clin Inform 2018; 9 (01) 99-104
- 3 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
- 4 Moore C, Valenti A, Robinson E, Perkins R. Using log data to measure provider EHR activity at a cancer center during rapid telemedicine deployment. Appl Clin Inform 2021; 12 (03) 629-636
- 5 Womack DM, Hribar MR, Steege LM, Vuckovic NH, Eldredge DH, Gorman PN. Registered nurse strain detection using ambient data: an exploratory study of underutilized operational data streams in the hospital workplace. Appl Clin Inform 2020; 11 (04) 598-605
- 6 Melnick ER, West CP, Nath B. et al. The association between perceived electronic health record usability and professional burnout among US nurses. J Am Med Inform Assoc 2021; 28 (08) 1632-1641
- 7 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
- 8 Tajirian T, Stergiopoulos V, Strudwick G. et al. The influence of electronic health record use on physician burnout: cross-sectional survey. J Med Internet Res 2020; 22 (07) e19274
- 9 Linzer M, Poplau S, Babbott S. et al. Worklife and wellness in academic general internal medicine: results from a national survey. J Gen Intern Med 2016; 31 (09) 1004-1010
- 10 Eschenroeder HC, Manzione LC, Adler-Milstein J. et al. Associations of physician burnout with organizational electronic health record support and after-hours charting. J Am Med Inform Assoc 2021; 28 (05) 960-966
- 11 Overhage JM, McCallie Jr D. Physician time spent using the electronic health record during outpatient encounters: a descriptive study. Ann Intern Med 2020; 172 (03) 169-174
- 12 Overhage JM, Johnson KB. Pediatrician electronic health record time use for outpatient encounters. Pediatrics 2020; 146 (06) e20194017
- 13 Sockalingam S, Tavares W, Charow R. et al. Examining associations between physician data utilization for practice improvement and lifelong learning. J Contin Educ Health Prof 2019; 39 (04) 236-242
- 14 Sieja A, Kim E, Holmstrom H. et al. Multidisciplinary sprint program achieved specialty-specific EHR optimization in 20 clinics. Appl Clin Inform 2021; 12 (02) 329-339
- 15 Otokiti AU, Craven CK, Shetreat-Klein A, Cohen S, Darrow B. Beyond Getting Rid of Stupid Stuff in the Electronic Health Record (Beyond-GROSS): protocol for a user-centered, mixed-method intervention to improve the electronic health record system. JMIR Res Protoc 2021; 10 (03) e25148
- 16 Sequeira L, Almilaji K, Strudwick G, Jankowicz D, Tajirian T. EHR “SWAT” teams: a physician engagement initiative to improve electronic health record (EHR) experiences and mitigate possible causes of EHR-related burnout. JAMIA Open 2021; 4 (02) ooab018
- 17 Kannampallil T, Abraham J, Lou SS, Payne PRO. Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects. J Am Med Inform Assoc 2021; 28 (05) 1032-1037
- 18 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: a new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
- 19 Cox ML, Farjat AE, Risoli TJ. et al. Documenting or operating: where is time spent in general surgery residency?. J Surg Educ 2018; 75 (06) e97-e106
- 20 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One 2019; 14 (02) e0205379
- 21 Mishra P, Kiang JC, Grant RW. Association of medical scribes in primary care with physician workflow and patient experience. JAMA Intern Med 2018; 178 (11) 1467-1472
- 22 Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019; 2019: 136-144
- 23 Weiner JP, Fowles JB, Chan KS. New paradigms for measuring clinical performance using electronic health records. Int J Qual Health Care 2012; 24 (03) 200-205
- 24 Kanger C, Brown L, Mukherjee S, Xin H, Diana ML, Khurshid A. Evaluating the reliability of EHR-generated clinical outcomes reports: a case study. EGEMS (Wash DC) 2014; 2 (03) 1102
- 25 Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (03) 480-490
- 26 TimeCaT. TimeCaT 3.9. Accessed July 1, 2021 at: https://lopetegui.net/timecat/39/login/
- 27 Read-Brown S, Hribar MR, Reznick LG. et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
- 28 Tai-Seale M, Olson CW, Li J. et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
- 29 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
- 30 Cooley J, Smith S. Privacy-preserving screen capture: towards closing the loop for health IT usability. J Biomed Inform 2013; 46 (04) 721-733
- 31 UserFocus. Log usability tests like a pro. Accessed May 31, 2021 at: https://www.userfocus.co.uk/articles/datalogging.html
- 32 Sinha A, Stevens LA, Su F, Pageler NM, Tawfik DS. Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021; 12 (04) 737-744
- 33 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. 2015 AMIA Annual Symposium Proceedings. San Francisco, CA, November 14–18, 2015: 1909-1917
- 34 Wolpert M, Rutter H. Using flawed, uncertain, proximate and sparse (FUPS) data in the context of complexity: learning from the case of child mental health. BMC Med 2018; 16 (01) 82