Identifying Electronic Health Record Tasks and Activity Using Computer Vision

Abstract

Background

Time spent in the electronic health record (EHR) is an important measure of clinical activity. Vendor-derived EHR use metrics may not correspond to actual EHR experience. Raw EHR audit logs enable customized EHR use metrics, but translating discrete timestamps to time intervals is challenging. There are insufficient data available to quantify inactivity between audit log timestamps.

Objective

This study aimed to develop and validate a computer vision-based model that (1) classifies EHR tasks and identifies task changes and (2) quantifies active-use time from clinician session screen recordings of EHR use. This study also aimed to develop and validate a computer vision-based model that (1) classifies EHR tasks and identifies task changes and (2) quantifies active-use time from clinician session screen recordings of EHR use.

We generated 111 minutes of simulated workflow in an Epic sandbox environment for development and training and collected 86 minutes of real-world clinician session recordings for validation. The model used YOLOv8, Tesseract OCR, and a predefined dictionary to perform task classification and task change detection. We developed a frame comparison algorithm to delineate activity from inactivity and thus quantify active time. We compared the model's output of task classification, task change identification, and active time quantification against clinician annotations. We then performed a post hoc sensitivity analysis to identify the model's accuracy when using optimal parameters.

Results

Our model classified time spent in various high-level tasks with 94% accuracy. It detected task changes with 90.6% sensitivity. Active-use quantification varied by task, with lower mean absolute percentage error (MAPE) for tasks with clear visual changes (e.g., Results Review) and higher MAPE for tasks with subtle interactions (e.g., Note Entry). A post hoc sensitivity analysis revealed improvement in active-use quantification with a lower threshold of inactivity than initially used.

Conclusion

A computer vision approach to identifying tasks performed and measuring time spent in the EHR is feasible. Future work should refine task-specific thresholds and validate across diverse settings. This approach enables defining optimal context-sensitive thresholds for quantifying clinically relevant active EHR time using raw audit log data.

Keywords

electronic health records - neural network - pediatrics

Protection of Human and Animal Subjects

This study was approved by Stanford University's IRB with a waiver of informed consent. The training phase of this study exclusively used Epic's Sandbox Instance, containing only simulated patient data. The validation phase employed technical measures to protect provider and patient privacy, including random selection of providers and sessions, automated obscuring of non-Epic applications through the Citrix Session Recorder, and task classification that sanitized any sensitive information. No animal subjects were involved in this research.

Publication History

Received: 02 June 2025

Accepted: 08 September 2025

Accepted Manuscript online:
10 September 2025

Article published online:
17 October 2025

• References

• 1 Ye J. The impact of electronic health record-integrated patient-generated health data on clinician burnout. J Am Med Inform Assoc 2021; 28 (05) 1051-1056
• 2 Kroth PJ, Morioka-Douglas N, Veres S. et al. The electronic elephant in the room: Physicians and the electronic health record. JAMIA Open 2018; 1 (01) 49-56
• 3 Baxter SL, Apathy NC, Cross DA, Sinsky C, Hribar MR. Measures of electronic health record use in outpatient settings across vendors. J Am Med Inform Assoc 2021; 28 (05) 955-959
• 4 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609
• 5 Kannampallil T, Abraham J, Lou SS, Payne PRO. Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects. J Am Med Inform Assoc 2021; 28 (05) 1032-1037
• 6 McPeek-Hinz E, Boazak M, Sexton JB. et al. Clinician burnout associated with sex, clinician type, work culture, and use of electronic health records. JAMA Netw Open 2021; 4 (04) e215686
• 7 Dymek C, Kim B, Melton GB, Payne TH, Singh H, Hsiao CJ. Building the evidence-base to reduce electronic health record-related clinician burden. J Am Med Inform Assoc 2021; 28 (05) 1057-1061
• 8 Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020; 27 (04) 531-538
• 9 Sieja A, Markley K, Pell J. et al. Optimization sprints: Improving clinician satisfaction and teamwork by rapidly reducing electronic health record burden. Mayo Clin Proc 2019; 94 (05) 793-802
• 10 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 11 Tawfik DS, Sinha A, Bayati M. et al. Frustration with technology and its relation to emotional exhaustion among health care workers: Cross-sectional observational study. J Med Internet Res 2021; 23 (07) e26817
• 12 Rule A, Kannampallil T, Hribar MR. et al. Guidance for reporting analyses of metadata on electronic health record use. J Am Med Inform Assoc 2024; 31 (03) 784-789
• 13 Hron JD, Lourie E. Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency. J Am Med Inform Assoc 2020; 27 (04) 644-646
• 14 Sinha A, Stevens LA, Su F, Pageler NM, Tawfik DS. Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021; 12 (04) 737-744
• 15 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
• 16 Rule A, Melnick ER, Apathy NC. Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures. J Am Med Inform Assoc 2022; 30 (01) 144-154
• 17 Sinsky C, Colligan L, Li L. et al. Allocation of physician time in ambulatory practice: A time and motion study in 4 specialties. Ann Intern Med 2016; 165 (11) 753-760
• 18 Rotenstein LS, Holmgren AJ, Downing NL, Longhurst CA, Bates DW. Differences in clinician electronic health record use across adult and pediatric primary care specialties. JAMA Netw Open 2021; 4 (07) e2116375
• 19 Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (03) 480-490
• 20 Kannampallil T, Adler-Milstein J. Using electronic health record audit log data for research: insights from early efforts. J Am Med Inform Assoc 2022; 30 (01) 167-171
• 21 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: A new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
• 22 Magon HS, Helkey D, Shanafelt T, Tawfik D. Creating conversion factors from EHR event log data: A comparison of investigator-derived and vendor-derived metrics for primary care physicians. AMIA Annu Symp Proc 2024; 2023: 1115-1124
• 23 Swanberg SM. Resource review: ActivePresenter v9. J Med Libr Assoc 2023; 111 (04) 854-857
• 24 Varghese R, Sambath M. YOLOv8: A Novel Object Detection Algorithm with Enhanced Performance and Robustness. Paper presented at: 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India. IEEE; 2024 :pp. 1–6.
• 25 Labelbox. Labelbox. 2023 . Accessed September 11, 2025 at: https://labelbox.com
• 26 Smith R. An Overview of the Tesseract OCR Engine. Paper presented at: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil. IEEE; 2007 :Vol 2, pp. 629–633.