Download PDF
Appl Clin Inform 2022; 13(05): 928-934
DOI: 10.1055/s-0042-1756424
Case Report

Accuracy of Physician Electronic Health Record Usage Analytics using Clinical Test Cases

Brian Lo   *
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
2   Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada
3   Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
,
Lydia Sequeira   *
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
2   Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada
3   Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
,
Gillian Strudwick
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
2   Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada
3   Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
,
Damian Jankowicz
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
,
Khaled Almilaji
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
,
Anjchuca Karunaithas
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
4   Department of Health and Society, University of Toronto Scarborough, Scarborough, Canada
,
Dennis Hang
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
5   Health Information Science, University of Victoria, Victoria, British Columbia, Canada
,
Tania Tajirian
1   Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada
6   Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
› Author Affiliations Funding There was no formal source of funding for this work. In-kind support was provided by the Centre for Addiction and Mental Health.
› Further Information
    Permissions and Reprints

Abstract

Usage log data are an important data source for characterizing the potential burden related to use of the electronic health record (EHR) system. However, the utility of this data source has been hindered by concerns related to the real-world validity and accuracy of the data. While time–motion studies have historically been used to address this concern, the restrictions caused by the pandemic have made it difficult to carry out these studies in-person. In this regard, we introduce a practical approach for conducting validation studies for usage log data in a controlled environment. By developing test runs based on clinical workflows and conducting them within a test EHR environment, it allows for both comparison of the recorded timings and retrospective investigation of any discrepancies. In this case report, we describe the utility of this approach for validating our physician EHR usage logs at a large academic teaching mental health hospital in Canada. A total of 10 test runs were conducted across 3 days to validate 8 EHR usage log metrics, finding differences between recorded measurements and the usage analytics platform ranging from 9 to 60%.

Keywords

analytics - documentation burden - EHR system - clinical informatics

Data Availability Statement

The validation data from this project can be shared upon reasonable request to the corresponding author.


Protection of Human and Animal Subjects

The study was reviewed by CAMH Quality Project Ethics Review Board.


* These authors contributed equally.


Supplementary Material



Publication History

Received: 06 May 2022

Accepted: 25 July 2022

Article published online:
05 October 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Moy AJ, Schwartz JM, Chen R. et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. J Am Med Inform Assoc 2021; 28 (05) 998-1008
    CrossrefPubMedGoogle Scholar
  • 2 Kannampallil TG, Denton CA, Shapiro JS, Patel VL. Efficiency of emergency physicians: insights from an observational study using EHR log files. Appl Clin Inform 2018; 9 (01) 99-104
    Article in Thieme ConnectPubMedGoogle Scholar
  • 3 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
    CrossrefPubMedGoogle Scholar
  • 4 Moore C, Valenti A, Robinson E, Perkins R. Using log data to measure provider EHR activity at a cancer center during rapid telemedicine deployment. Appl Clin Inform 2021; 12 (03) 629-636
    Article in Thieme ConnectPubMedGoogle Scholar
  • 5 Womack DM, Hribar MR, Steege LM, Vuckovic NH, Eldredge DH, Gorman PN. Registered nurse strain detection using ambient data: an exploratory study of underutilized operational data streams in the hospital workplace. Appl Clin Inform 2020; 11 (04) 598-605
    Article in Thieme ConnectPubMedGoogle Scholar
  • 6 Melnick ER, West CP, Nath B. et al. The association between perceived electronic health record usability and professional burnout among US nurses. J Am Med Inform Assoc 2021; 28 (08) 1632-1641
    CrossrefPubMedGoogle Scholar
  • 7 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
    CrossrefPubMedGoogle Scholar
  • 8 Tajirian T, Stergiopoulos V, Strudwick G. et al. The influence of electronic health record use on physician burnout: cross-sectional survey. J Med Internet Res 2020; 22 (07) e19274
    CrossrefPubMedGoogle Scholar
  • 9 Linzer M, Poplau S, Babbott S. et al. Worklife and wellness in academic general internal medicine: results from a national survey. J Gen Intern Med 2016; 31 (09) 1004-1010
    CrossrefPubMedGoogle Scholar
  • 10 Eschenroeder HC, Manzione LC, Adler-Milstein J. et al. Associations of physician burnout with organizational electronic health record support and after-hours charting. J Am Med Inform Assoc 2021; 28 (05) 960-966
    CrossrefPubMedGoogle Scholar
  • 11 Overhage JM, McCallie Jr D. Physician time spent using the electronic health record during outpatient encounters: a descriptive study. Ann Intern Med 2020; 172 (03) 169-174
    CrossrefPubMedGoogle Scholar
  • 12 Overhage JM, Johnson KB. Pediatrician electronic health record time use for outpatient encounters. Pediatrics 2020; 146 (06) e20194017
    CrossrefPubMedGoogle Scholar
  • 13 Sockalingam S, Tavares W, Charow R. et al. Examining associations between physician data utilization for practice improvement and lifelong learning. J Contin Educ Health Prof 2019; 39 (04) 236-242
    CrossrefPubMedGoogle Scholar
  • 14 Sieja A, Kim E, Holmstrom H. et al. Multidisciplinary sprint program achieved specialty-specific EHR optimization in 20 clinics. Appl Clin Inform 2021; 12 (02) 329-339
    Article in Thieme ConnectPubMedGoogle Scholar
  • 15 Otokiti AU, Craven CK, Shetreat-Klein A, Cohen S, Darrow B. Beyond Getting Rid of Stupid Stuff in the Electronic Health Record (Beyond-GROSS): protocol for a user-centered, mixed-method intervention to improve the electronic health record system. JMIR Res Protoc 2021; 10 (03) e25148
    CrossrefPubMedGoogle Scholar
  • 16 Sequeira L, Almilaji K, Strudwick G, Jankowicz D, Tajirian T. EHR “SWAT” teams: a physician engagement initiative to improve electronic health record (EHR) experiences and mitigate possible causes of EHR-related burnout. JAMIA Open 2021; 4 (02) ooab018
    CrossrefPubMedGoogle Scholar
  • 17 Kannampallil T, Abraham J, Lou SS, Payne PRO. Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects. J Am Med Inform Assoc 2021; 28 (05) 1032-1037
    CrossrefPubMedGoogle Scholar
  • 18 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: a new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
    CrossrefPubMedGoogle Scholar
  • 19 Cox ML, Farjat AE, Risoli TJ. et al. Documenting or operating: where is time spent in general surgery residency?. J Surg Educ 2018; 75 (06) e97-e106
    CrossrefPubMedGoogle Scholar
  • 20 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One 2019; 14 (02) e0205379
    CrossrefPubMedGoogle Scholar
  • 21 Mishra P, Kiang JC, Grant RW. Association of medical scribes in primary care with physician workflow and patient experience. JAMA Intern Med 2018; 178 (11) 1467-1472
    CrossrefPubMedGoogle Scholar
  • 22 Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019; 2019: 136-144
    PubMedGoogle Scholar
  • 23 Weiner JP, Fowles JB, Chan KS. New paradigms for measuring clinical performance using electronic health records. Int J Qual Health Care 2012; 24 (03) 200-205
    CrossrefPubMedGoogle Scholar
  • 24 Kanger C, Brown L, Mukherjee S, Xin H, Diana ML, Khurshid A. Evaluating the reliability of EHR-generated clinical outcomes reports: a case study. EGEMS (Wash DC) 2014; 2 (03) 1102
    PubMedGoogle Scholar
  • 25 Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (03) 480-490
    CrossrefPubMedGoogle Scholar
  • 26 TimeCaT. TimeCaT 3.9. Accessed July 1, 2021 at: https://lopetegui.net/timecat/39/login/
    PubMed
  • 27 Read-Brown S, Hribar MR, Reznick LG. et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
    CrossrefPubMedGoogle Scholar
  • 28 Tai-Seale M, Olson CW, Li J. et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
    CrossrefPubMedGoogle Scholar
  • 29 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
    CrossrefPubMedGoogle Scholar
  • 30 Cooley J, Smith S. Privacy-preserving screen capture: towards closing the loop for health IT usability. J Biomed Inform 2013; 46 (04) 721-733
    CrossrefPubMedGoogle Scholar
  • 31 UserFocus. Log usability tests like a pro. Accessed May 31, 2021 at: https://www.userfocus.co.uk/articles/datalogging.html
    PubMed
  • 32 Sinha A, Stevens LA, Su F, Pageler NM, Tawfik DS. Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021; 12 (04) 737-744
    Article in Thieme ConnectPubMedGoogle Scholar
  • 33 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. 2015 AMIA Annual Symposium Proceedings. San Francisco, CA, November 14–18, 2015: 1909-1917
    Google Scholar
  • 34 Wolpert M, Rutter H. Using flawed, uncertain, proximate and sparse (FUPS) data in the context of complexity: learning from the case of child mental health. BMC Med 2018; 16 (01) 82
    CrossrefPubMedGoogle Scholar