Qualitative Verification of Machine Learning-Based Burnout Predictors in Primary Care Physicians: An Exploratory Study

Abstract

Background

Electronic health record (EHR) usage measures may quantify physician activity at scale and predict practice settings with a high risk for physician burnout, but their relation to experiences is poorly understood.

Objectives

This study aimed to explore the EHR-related experiences and well-being of primary care physicians in comparison to EHR usage measures identified as important for predicting burnout from a machine learning model.

Methods

Exploratory qualitative study with semi-structured interviews of primary care physicians and clinic managers from a large academic health system and its community physician partners. We included primary care clinics with high burnout scores, low burnout scores, or large changes in burnout scores between 2020 and 2022, relative to all primary care clinics in the health system. We conducted inductive and deductive coding of interview responses using a priori themes related to the machine learning model categories of patient load, documentation burden, messaging burden, orders, and physician distress and fulfillment.

Results

Interviews with 16 physicians and 4 clinic managers identified burdens related to three dominant themes: (1) messaging and documentation burdens are high and require more time than most physicians have available during standard working hours. (2) While EHR-related burdens are high they also provide patient-care benefits. (3) Turnover and insufficient staffing exacerbate time demands associated with patient load. Dimensions that are difficult to quantify, such as a perceived imbalance between job demands and individual resources, also contribute to burnout and were consistent across all themes.

Conclusion

EHR-related work burden, largely quantifiable through EHR usage measures, are major source of distress among primary care physicians. Organizational recognition of this work as well as staffing and support to predict associated work burden may increase professional fulfillment and reduce burnout among primary care physicians.

 

MissingLinkText

Keywords

physician burnout - machine learning - experiences - electronic health record usage measures

Background and Significance

Symptoms of burnout among physicians continue to rise, with recent reports suggesting that at the height of the pandemic burnout affected 68 to 76% of all primary care physicians.[1] [2] This finding has sobering implications for other aspects of physician well-being, quality of patient care, access to care, and institutional costs.[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] Identifying rates of burnout traditionally has been done through survey collection, although there are increasing efforts to identify at-risk work units using prediction models that leverage ubiquitously collected electronic health record (EHR) metadata.[16] [17] [18] What is still missing, however, is an understanding of the relationship between these quantitative EHR measures and the experiences of physicians. Without this knowledge, it is unknown whether there are additional features that could be measured to improve the precision and accuracy of prediction models.

Routinely collected EHR metadata include a myriad of measures related to aspects of the work environment known to be associated with burnout (e.g., measures related to workload, efficiency EHR usability, and impaired work-life integration).[9] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] Building upon this existing research, we recently developed a burnout prediction model for primary care physicians, which identified several EHR usage measures that were limited in predicting individual burnout scores but showed promise in identifying clinics at increased risk.[16] The top ten features of this model are outlined in [Table 1].

Table 1 Top 10 quantitative findings from the burnout prediction model, and related key concepts and themes

Feature importance rank	Quantitative finding	Concept	Theme
1	Higher automated/administrative messages per day predict higher burnout	Automated messages	Messaging and documentation burden
2	Higher number of characters written by team members predicts higher burnout	Progress notes	Messaging and documentation burden
3	Higher provider efficiency score predicts lower burnout	Usability	EHR-related burdens
4	Higher and lower year of birth predicts lower burnout	Personal factors	Personal and professional factors
5	Higher proportion of referral orders placed using user preferences predicts lower burnout	Usability	EHR-related burdens
6	Higher minutes spent responding to medical advice requests predict higher burnout	Expanded access	EHR-related burdens
		Message triaging	Messaging and documentation burden
7	Lower number of shared smart phrases predicts lower burnout	Progress notes	Messaging and documentation burden
8	Higher proportion of other orders placed using user preferences predicts lower burnout	Usability	EHR-related burdens
9	Higher minutes spent in notes per 8 h of scheduled appointments predicts lower burnout	Progress notes	Messaging and documentation burden
		Patients vs. systems	Turnover and insufficient staffing
10	Higher proportion of characters written by copy-paste predicts higher burnout	Progress notes	Messaging and documentation burden
		Time pressure	Turnover and insufficient staffing

 

MissingLinkText

Objectives

To assess the degree to which these prediction models reflect true sources of frustration for physicians, this study interviewed primary care physicians to: (1) understand the relationships between well-being and qualitative EHR-related experiences in comparison to quantitative EHR usage metrics, and (2) identify additional features relevant to well-being that may not adequately be captured in the current prediction model.

 

MissingLinkText

Methods

This exploratory qualitative study used semi-structured interviews to collect data from physicians and clinic managers who work in primary care clinics within a large academic medical center or one of its community affiliates. Results are reported using the COREQ criteria.[41]

Generation of Prediction Model and Interview Guide Development

A full description of the burnout survey and prediction modeling procedures were previously published.[16] Briefly, we used institutional well-being surveys from 2019 through 2022 to identify burnout symptoms among primary care physicians, using the Emotional Exhaustion and Interpersonal Disengagement scales of the Stanford Professional Fulfillment Index, which align with the established literature for this instrument.[42] We combined these well-being survey results from 2019 and 2020 with contemporaneous EHR usage measures and employed a gradient boosting machine to generate a burnout prediction model to identify EHR usage measures that most strongly influenced the model's output.

Using this data-informed model, we developed an interview guide based on the most influential EHR usage measures, with an emphasis on identifying aspects that physicians found fulfilling and challenging about their work in the following areas: (1) patient load, (2) notes, messages, and time management, (3) orders, and (4) burnout and well-being.

 

MissingLinkText

Clinic Selection and Recruitment

To inform targeted clinic recruitment using maximum variation sampling, we evaluated aggregate burnout scores by the clinic for the years 2020 and 2022. We identified the top five clinics (out of 18 candidates) in the following four categories: (1) highest average burnout scores in 2022, (2) lowest average burnout scores in 2022, (3) greatest increase in burnout scores between 2020 and 2022, and (4) greatest decrease in burnout scores between 2020 and 2022. This procedure resulted in 11 unique clinics and guided recruitment efforts. To mitigate any potential bias, study personnel involved in recruitment and interviewing were blinded to the classification of each clinic's inclusion. Physicians and managers (i.e., medical, clinic, and assistant clinic managers) were targeted to explore clinic-level factors impacting physicians' experiences. The top 6 of the 11 identified clinics were initially contacted via email. Three clinics out of the original six did not participate (two for lack of response and one for active decline), so an additional round of three clinics from clinics in high, low, and changing burnout groups were contacted.

 

MissingLinkText

Data Collection

Clinic managers were asked to forward recruitment material to physicians in their clinics. Potential participants were then contacted by email up to three times by a member of the research team (CB) and asked to complete a brief in-take survey to collect demographic information and indicate preferred scheduling for interviews. Interviews were conducted via video conferencing. Participation was voluntary and participants received a $50 incentive for participation. Since this project was deemed expedited with a waiver of documentation by the Stanford IRB (#49374), each participant reviewed a consent document with a researcher prior to the interview and gave verbal consent for recording.

Interviews were in-depth and semi-structured with a topic focus on patient load (complexity and stability of panel); EHR notes, messages, and orders (burden, workflows, and time demands); physician distress (experiences of burnout and desired interventions to prevent burnout) and fulfillment; and change over time. The full interview guide is provided as [Supplementary Material S1] (available in the online version only).

Interviews were conducted via Zoom (Zoom Video Communications, San Jose, California, United States), from February to June 2023 by a female member of the research team trained in qualitative research methods who had no prior relationship with the participants (CB). Interviews were audio recorded, de-identified, and transcribed verbatim by a professional transcription service (Rev, Austin, Texas, United States). Transcripts were analyzed by two members of the research team (CB-J and CB) using thematic analysis and consensus coding approaches[43] [44] [45] and qualitative analysis software (NVivo, QSR International, Burlington, Massachusetts, United States).

 

MissingLinkText

Data Analysis

Twenty out of 74 invited participants from 6 clinics were interviewed: 16 physicians, and 4 managers (medical managers and assistant clinic managers). The nonresponse rate was 73% from the six clinics that participated.

Two of the study investigators (CB and CB-J) met three times to review the data and develop and refine the codebook, provided in [Supplementary Material S1] (available in the online version only). A qualitative researcher (CB) coded deductively, identifying elements related to the verification and validation of the machine learning model, and inductively identifying themes within the data. This coding was subsequently reviewed jointly by CB and CB-J on three separate occasions and adjusted to consensus through several additional weekly discussions with the broader research team (DT, CB, CB-J, and MW) during the data analysis phase.[43] [44] [45] Our analytic approach also included a presentation of the findings with key informants (SSS, TS, and JP) and the institution's Physician Wellness Forum to explore whether interpretation patterns resonated with the perspectives of educational, wellness, and quality experts.

 

MissingLinkText

 

MissingLinkText

Results

Overview and Study Participants

We approached a total of nine primary care clinic managers representing five clinics with high or increasing burnout and four clinics with low or decreasing burnout. One clinic manager declined participation (citing a desire to protect their physicians from additional burden), two did not respond, and the participating six clinics represented two academic primary care clinics and four community affiliate clinics. Of these final six recruited clinics, three had high and increasing burnout scores, one had high stable burnout scores, one had low but increasing burnout scores, and one had low stable burnout scores. Participants consisted of 16 physicians and 4 clinic managers working in the 6 clinics and are identified using P#. Thirteen women and seven men participated.

 

MissingLinkText

Major Themes

Upon achieving thematic sufficiency, we identified five themes: Messaging/Documentation Burden, EHR-related Burdens, Turnover/Insufficient Staffing, Teamwork, and Destabilizing Change. The first three of these listed themes were more developed and aligned to the study question; therefore they are described in further detail here. Of note, messaging (between patients and the healthcare team, or among members of the healthcare team) and clinical documentation represent distinct workflows with different purposes but are presented within a single theme representing EHR-mediated communication.

Messaging and Documentation Burden

Physicians noted that large amounts of both clinic and after-hour time were spent dealing with In Basket management (the internal messaging system within the Epic EHR). One physician participant summarized this as “the bane of a primary care doctor's experience” and noted that one of the challenges with automated messages is they are a mixture of useful and not useful information ([Table 2]).

Table 2 Identified themes, related concepts, and representative quotes from interview participants, in relation to the quantitative burnout prediction model

Theme	Concept	Quote	Relation to the quantitative model	Comment
Messaging and documentation burden	Automated messages	“If one of my patients is in the emergency room, I'll get notified. If they see another physician, I'll get notified. So some of that stuff is really, really useful and I really want to know it and I want to know it immediately. But some of that stuff I don't need to know”	Concordant	Message value as determined by the physician may not correspond to message subtype as categorized by the EHR vendor
	Message triaging	“So I think medical assistants don't always have the training to be able to triage or deal with those [patient] messages appropriately. So even though we have support staff to go through the messages, the vast majority of it falls on the physician's plate”	Concordant	The value of a team-based message management approach is influenced by the training and capabilities of all involved members of the team
	Message urgency	“So the organization wanted us to send out depression screening questions […] And so that means that while I'm sitting trying to type my note at 9:00 p.m. I might get, ‘Oh, look that patient answered the questionnaire that they’re suicidal now' and now I have to deal with that without my staff. I'm not even in the office”	New insight	Message content may dictate both the urgency of response and the support resources needed by the physician to respond, independent of the message category
EHR-related burdens	Expanded access	“Yeah, the [EHR] is a pain. I think it's created a lot more access for our patients who have utilized it as another avenue for care and oftentimes for asynchronous free care. And it's created a lot of work”	Concordant	Technology-facilitated care may enable supply-demand imbalances, with healthcare demands outweighing health system resources
	Feature implementation	“[The institution] just opened this portal and literally never said, ‘Here, have some time in your day for this.’ The expectation was it's just another fire hose, deal with it. This is what patients want, or this is what we want, or I have no idea”	New insight	Response to practice changes may be influenced by physician participation in decision-making, transparent communication, and adequate support
Turnover and insufficient staffing	Patients vs. systems	“I think the last year or two, it's been a lot tougher. When I go home at 10 o'clock on Mondays, I love the patients, I don't mind working really hard for the patients. I just get so frustrated that it seems to be the system that's dragging me down”	Concordant	System inefficiencies may contribute to effort-reward imbalance
	Time pressure	“I've been so burned out. I hate going in to see patients every time. Every Sunday, I am so miserable. My kids know that Tuesday night is the best night because mommy doesn't have patient care again until Monday. That's terrible because when I'm in the visits with my patients, I like them. But it's something about the rush and all of the constraints that we have on us. So many expectations. So much of it is clerical”	Concordant	Chaotic environments and time pressure are partially attributable to administrative burden and are not fully alleviated by reduced work hours
	Turnover	“And the entire time I've been here, there has never been everybody filled and nobody leaving soon. … I don't have enough space for my patients to get in when they want … and that is, as a person who likes taking care of people, is extremely difficult”	New insight	Appointment availability may affect patient and physician satisfaction, with team instability a potential driving factor

Another participant stated, “I feel an overwhelming sense of dread as I look at the number of patient messages that I have.” Physicians tried to clear their In Basket each day, but were often unable to due to high message volume or other responsibilities, with one noting “by the time I get to my messages sometimes it's like midnight.” Many participants reported doing message management on their nonclinic days, with one stating, “I was recently on vacation for a week, and I logged in maybe three or four times to stay on top of it.”

Beyond the volume of messages, participants also noted that they were often ill-suited to deal with the content and urgency of messages, particularly when receiving messages at inopportune times or under scenarios in which they did not have adequate support and resources to respond. Additionally, they reported that clinic staff were inadequately prepared to reduce the message burden on physicians ([Table 2]). Even if weekly time was allotted for messaging (In Basket) work, the volume and urgency of the messages precluded physicians from handling them only during those times, with one physician suggesting, “I would like to have time built into our schedules for the In Basket work on a daily basis … We need to have an hour built into any day just to do that In Basket.”

When documenting visits, some participants expressed that they preferred self-written notes because they had concerns that relying on an assistant or scribe would slow workflows due to misunderstandings and required training time. In addition, scribes were not always available for physicians who preferred them. Several physicians also expressed that helping complex patients (i.e., those requiring more time in the EHR to accurately capture the presenting concern) provided high professional fulfillment.

Other physicians highlighted a potential discrepancy between EHR use measures and their experience. Several physicians stated that they prefer (when appointment time allows) to talk through their thought process with patients while they go through the process of writing notes or placing orders, resulting in longer measured EHR active use time, but also higher professional fulfillment and quality patient care.

Several participants offered potential solutions to the challenges they noted, including qualified staff and dedicated time for messaging (In Basket) management: “… they're doing things like they're hiring an RN, which is wonderful because RNs are not cheap, to try to offload us in the inbox. That's helpful. Every little bit is helpful.” and “I would like to have time built into our schedules for the In Basket work on a daily basis. We need that.”

 

MissingLinkText

EHR-Related Burdens

Several physicians expressed mixed feelings when describing the tension between the benefits of the EHR and the additional burdens it provides. Physicians appreciated the accessibility and organization of the EHR compared with previous systems, but strongly disliked how much time certain functions required.

In particular, physicians commented on the ability of the EHR to expand access to patients, particularly during the COVID-19 pandemic, with many expressing that the way in which it was implemented increased physician burden ([Table 2]). Implementation without adequate support appeared to add to EHR-related burdens for primary care physicians and managers.

 

MissingLinkText

Turnover and Insufficient Staffing

Interview participants expressed that although they continue to enjoy caring for individual patients, they were overwhelmed by the number of new patients due to physician turnover particularly when the new patients were complex. One physician reflected on the increase in patients and messages since the COVID pandemic as a tension between patients and systems ([Table 2]). Another noted that excess patient load due to physician and staff turnover exacerbated the system aspect of managing patient load and demand ([Table 2]).

Although none of the participants identified patients themselves as a cause of burnout, chaotic system pressures, and unrealistic expectations were identified as contributors ([Table 2]). Several physicians expressed that healthcare provider team familiarity and clinician-patient continuity affected their ability to work efficiently and find professional fulfillment. One participant noted, “I think I see some providers really struggling with the excess burden, not only of inbox, but I think with the churn of the staff,” while another described it as, “… if my nurse is out on leave, then there are other nurses that are already assigned to pods that are covering us in addition. So, it is not the same triage or depth of triage that they can provide.”

Conversely, participants noted that increasing staff familiarity and experience could be particularly important for them to assist with message triaging and order preparation, “I feel like if we could somehow fix this issue of staff turnover, I think that would definitely help the workflow overall.”

 

MissingLinkText

Demands and Resources Alignment

Across the identified themes, several participants reflected that organizational demands do not align with their available resources and felt that their agency was limited, thereby contributing to poor professional fulfillment. Many expressed that the organization seemed to increase job demands by increasing patient appointments or enabling patient messaging to increase patient satisfaction, but this is often done without providing protected time or flexibility in these activities. Several physicians reminisced about this change stating: “we used to have flexibility in our schedule” or “we used to have an hour each half session for documenting, for calling, for doing refills, for doing all that. Now I'm lucky if I have my lunch hour.”

Although workload and time demands were frequently cited as burdensome, the burden was never attributed to the patients themselves. Even physicians who self-identified as experiencing burnout did not express symptoms of depersonalization directed toward their patients. In fact, several participants reflected that they continued to care for and enjoy their interactions with patients in the midst of experiencing excessive burden.

 

MissingLinkText

 

MissingLinkText

Clinic Comparisons

Unblinded comparisons among clinics showed largely similar responses despite clinic-level burnout scores, with two key exceptions. Physicians who noted value misalignment as a source of distress represented the three clinics with high and increasing burnout scores; this was not observed with the other clinics. Physicians from the clinic with low stable burnout scores reported that their clinic provided a dedicated remote day of work to alleviate additional EHR burden, which was perceived as highly beneficial.

 

MissingLinkText

 

MissingLinkText

Discussion

In this study of primary care physicians, we discovered congruence between participant qualitative responses and the EHR audit log parameters identified to predict burnout at the clinic level through a recently developed machine learning model. Participants reported a high degree of work burden related to messaging, challenges maximizing team-based care during times of heightened turnover, and overall high demands for physician time that limit flexibility. In addition, several participants described an imbalance between organizational demands and individual resources that were perceived to drive the movement toward increased burdens that physicians did not find professionally fulfilling. Strikingly, participant responses centered primarily around institutional factors, consistent with our findings that the quantitative model carried greater predictive power for identifying clinic-level burnout risk, and the conceptualization of burnout as a work-related phenomenon.

The majority of participants described the EHR message (In Basket) management as a particular source of distress, consistent with our prediction model finding that the number of automated/administrative messages per day was the feature with the highest importance for burnout symptom prediction. These findings align with a growing body of evidence on the amount of messaging burden and its associations with burnout.[21] [26] [46] [47] [48] Notably, respondents expressed concerns with the implementation of and support for In Basket work rather than the concept itself. In particular, complaints centered around (1) the volume of messages and the relative infrequency of important content, (2) the fact that messages continue to pile up even while seeing other patients or out of the clinic, (3) limited support from other staff who can meaningfully reduce the burden of the messages, and (4) misaligned expectations and use of messaging by patients, particularly during and following the COVID-19 pandemic. These findings highlight the importance of organizational leadership to carefully consider the balance between burdens and resources as a contributor to individual physician well-being and organizational health.

Team-based care has frequently been championed as a solution to many of the excess demands perceived by primary care physicians.[49] [50] [51] [52] [53] [54] [55] [56] However, participants expressed several challenges to implementing effective team-based care, potentially explaining the seemingly counterintuitive finding of our prediction model in which note composition by team members is associated with higher burnout. In clinics where a team-based approach to patient care is employed, individual team members are not always educated or ideally suited to perform the tasks assigned to them. Regarding offloading excess messaging burden, several participants expressed that they (1) lacked dedicated support within their clinic, (2) had inconsistency in the individuals supporting them, or (3) had support teams that were misaligned with the tasks assigned to them (e.g., medical assistants triaging patient medical advice requests rather than registered nurses). These findings highlight the importance of maximizing team-based coordination and minimizing turnover as strategies to mitigate burnout.

However, participants expressed differing opinions on team-based documentation, with some stating that they preferred to write their own notes because they felt their notes were more efficient and accurate that way. Individual clinics may benefit from assessing their own teams to determine which of all possible interventions would be most high yield, including technology-based solutions such as ambient scribes.[57]

Underlying the specific domains discussed by participants we noted two underlying aspects: control and change. Several physicians noted a lack of control as a particular burden, especially as related to scheduling clinic hours, balancing patient visits with other more clerical tasks, and determining how the quality of their care was measured—all of which were described as decisions imposed upon them by the system and are related to well-described threats to well-being and retention.[58] [59] [60] Related to this, many of the burdens were framed within the context of change from a prior state which was often viewed in a more favorable light. The most common example of this related to the increased volume of patient medical advice requests, although other expressed sources of undesired change included physician and staff turnover, and balance between in-person and telehealth visits. These findings highlight the importance of physician inclusion and organizational communication in practice modification decisions.

In aggregate, these findings provide valuable updates to the conceptual framework underlying the quantitative prediction model.[16] Several of the identified system and physician experience factors were reinforced and highlighted by study participants, including workflow, payment structure (particularly in relation to medical advice requests), teamwork culture, job demands, efficiency of practice, time allocation, administrative work, perceived inefficiency, and work-home conflict. Notable additions to the conceptual framework include a feedback loop wherein physician turnover exacerbates several system factors contributing to a chaotic environment and a feedback loop in which patient dissatisfaction (particularly in relation to scheduling availability) exacerbates physicians' professional dissatisfaction.

Limitations

Study findings should be viewed in relation to the design. Although we sampled a variety of physicians and clinic managers from academic and community-based primary care clinics, experiences may differ for physicians who were invited but chose not to respond, whose clinic managers declined participation, or who work in other settings. Although the interviewers were not in hospital leadership roles, social desirability bias may have affected participants' responses. Interviews were conducted shortly after the intensive phase of the COVID-19 response, and ongoing practice transformation across the healthcare sector may shed additional insights about the evolving state of physician burden.

 

MissingLinkText

 

MissingLinkText

Conclusion

Many EHR-related burdens quantifiable through EHR usage measures correspond with the experiences of primary care physicians, who reported burdens related to messaging, team-based care, inadequate time to complete work demands, and frequent changes. Developing and including additional related EHR usage measures into future quantitative machine learning approaches may improve predictive performance and inform directed interventions to reduce burnout risk.

 

MissingLinkText

Clinical Relevance Statement

Primary care physicians report mixed feelings regarding the electronic health record (EHR) system, with notable benefits in addition to frequent burdens related to excessive messaging burden in comparison to time available, high note burden, ordering frustration, and high patient loads. Inefficiencies of teamwork, staff turnover, and personal-organizational values misalignment contributed to perceptions of frustration. EHR usage measures align with experiences and may quantify misalignment between demands and resources within primary care.

 

MissingLinkText

Multiple-Choice Questions

1. Which of the following was a frequently cited burden by primary care physicians?

   • System latency

   • In Basket and documentation burden

   • Interoperability and communication with other health systems

   • Patient complexity

   Correct Answer: The correct answer is option b. In Basket and documentation burden. Primary care physicians frequently cited In Basket as a burden, with comments centering around the volume of messages received, the content of messages that they were not always equipped to deal with, and the continuous nature of new messages resulting in In Basket work continuing during nonclinic days and vacations. These findings are consistent with quantitative approaches, which found the volume of automated In Basket messages to be one of the strongest predictors of primary care physician burnout.

2. When implementing a new feature to expand access to care for patients, which of the following was identified by primary care physicians as an important but often overlooked consideration?

   • Advertising the new feature to all patients equitably.

   • Pilot testing the new feature with patients to improve usability.

   • Ensuring the institution provides physicians with adequate resources to support the new feature.

   • Integrating artificial intelligence capabilities in the new feature.

   Correct Answer: The correct answer is option c. Ensuring the institution provides physicians with adequate resources to support the new feature. Particularly for features such as patient portals that improve access to care, many physicians noted that although the improved access to care was helpful to the patients, the institution did not often provide adequate resources of protected time or staffing to respond to the increased demands. This increased demand without increased resources was perceived to promote an imbalance and increase the risk of burnout.

 

MissingLinkText

 

MissingLinkText

Conflict of Interest

T.S. is co-inventor of the Well-being Index instruments (Physician Well-being Index, Nurse Well-being Index, Medical Student Well-being Index, and the Well-being Index) and the Mayo Leadership Index. Mayo Clinic holds the copyright for these instruments and has licensed them for use outside of Mayo Clinic. He receives a portion of any royalties received. As an expert on the well-being of healthcare professionals, he frequently gives grand rounds/keynote lecture presentations and provides advice for healthcare organizations. He receives honoraria for some of these activities.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was approved by the Institutional Review Board of Stanford University (protocol #49374).

• References

• 1 Shanafelt TD, West CP, Dyrbye LN. et al. Changes in burnout and satisfaction with work-life integration in physicians during the first 2 years of the COVID-19 pandemic. Mayo Clin Proc 2022; 97 (12) 2248-2258
• 2 Sexton JB, Adair KC, Proulx J. et al. Emotional exhaustion among US health care workers before and during the COVID-19 pandemic, 2019-2021. JAMA Netw Open 2022; 5 (09) e2232748
• 3 Tawfik DS, Profit J, Morgenthaler TI. et al. Physician burnout, well-being, and work unit safety grades in relationship to reported medical errors. Mayo Clin Proc 2018; 93 (11) 1571-1580
• 4 Tawfik DS, Scheid A, Profit J. et al. Evidence relating health care provider burnout and quality of care: a systematic review and meta-analysis. Ann Intern Med 2019; 171 (08) 555-567
• 5 Shanafelt T, Goh J, Sinsky C. The business case for investing in physician well-being. JAMA Intern Med 2017; 177 (12) 1826-1832
• 6 Brown SD, Goske MJ, Johnson CM. Beyond substance abuse: stress, burnout, and depression as causes of physician impairment and disruptive behavior. J Am Coll Radiol 2009; 6 (07) 479-485
• 7 Fahrenkopf AM, Sectish TC, Barger LK. et al. Rates of medication errors among depressed and burnt out residents: prospective cohort study. BMJ 2008; 336 (7642) 488-491
• 8 Hall LH, Johnson J, Watt I, Tsipa A, O'Connor DB. Healthcare staff wellbeing, burnout, and patient safety: a systematic review. PLoS One 2016; 11 (07) e0159015
• 9 Linzer M, Manwell LB, Williams ES. et al; MEMO (Minimizing Error, Maximizing Outcome) Investigators. Working conditions in primary care: physician reactions and care quality. Ann Intern Med 2009; 151 (01) 28-36 , W6-9
• 10 Lu DW, Dresden S, McCloskey C, Branzetti J, Gisondi MA. Impact of burnout on self-reported patient care among emergency physicians. West J Emerg Med 2015; 16 (07) 996-1001
• 11 Shanafelt TD, Bradley KA, Wipf JE, Back AL. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med 2002; 136 (05) 358-367
• 12 Tawfik DS, Sexton JB, Kan P. et al. Burnout in the neonatal intensive care unit and its relation to healthcare-associated infections. J Perinatol 2017; 37 (03) 315-320
• 13 Weigl M, Schneider A, Hoffmann F, Angerer P. Work stress, burnout, and perceived quality of care: a cross-sectional study among hospital pediatricians. Eur J Pediatr 2015; 174 (09) 1237-1246
• 14 Wen J, Cheng Y, Hu X, Yuan P, Hao T, Shi Y. Workload, burnout, and medical mistakes among physicians in China: a cross-sectional study. Biosci Trends 2016; 10 (01) 27-33
• 15 West CP, Tan AD, Habermann TM, Sloan JA, Shanafelt TD. Association of resident fatigue and distress with perceived medical errors. JAMA 2009; 302 (12) 1294-1300
• 16 Tawfik D, Bayati M, Liu J. et al. Predicting primary care physician burnout from electronic health record use measures. Mayo Clin Proc 2024; 99 (09) 1411-1421
• 17 Liu H, Lou SS, Warner BC, Harford DR, Kannampallil T, Lu C. HiPAL: A Deep Framework for Physician Burnout Prediction Using Activity Logs in Electronic Health Records. Presented at: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; 2022 ; Washington DC, USA. https://doi.org/10.1145/3534678.3539056
• 18 Lou SS, Liu H, Warner BC, Harford D, Lu C, Kannampallil T. Predicting physician burnout using clinical activity logs: model performance and lessons learned. J Biomed Inform 2022; 127: 104015
• 19 Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020; 27 (04) 531-538
• 20 Babbott S, Manwell LB, Brown R. et al. Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc 2014; 21 (e1): e100-e106
• 21 Dyrbye LN, Gordon J, O'Horo J. et al. Relationships between EHR-based audit log data and physician burnout and clinical practice process measures. Mayo Clin Proc 2023; 98 (03) 398-409
• 22 Dyrbye LN, Sotile W, Boone S. et al. A survey of U.S. physicians and their partners regarding the impact of work-home conflict. J Gen Intern Med 2014; 29 (01) 155-161
• 23 Eschenroeder HC, Manzione LC, Adler-Milstein J. et al. Associations of physician burnout with organizational electronic health record support and after-hours charting. J Am Med Inform Assoc 2021; 28 (05) 960-966
• 24 Gardner RL, Cooper E, Haskell J. et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019; 26 (02) 106-114
• 25 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 26 Hilliard RW, Haskell J, Gardner RL. Are specific elements of electronic health record use associated with clinician burnout more than others?. J Am Med Inform Assoc 2020; 27 (09) 1401-1410
• 27 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609
• 28 Linzer M, Poplau S, Grossman E. et al. A cluster randomized trial of interventions to improve work conditions and clinician burnout in primary care: results from the healthy work place (HWP) study. J Gen Intern Med 2015; 30 (08) 1105-1111
• 29 McPeek-Hinz E, Boazak M, Sexton JB. et al. Clinician burnout associated with sex, clinician type, work culture, and use of electronic health records. JAMA Netw Open 2021; 4 (04) e215686
• 30 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 31 Peccoralo LA, Kaplan CA, Pietrzak RH, Charney DS, Ripp JA. The impact of time spent on the electronic health record after work and of clerical work on burnout among clinical faculty. J Am Med Inform Assoc 2021; 28 (05) 938-947
• 32 Rassolian M, Peterson LE, Fang B. et al. Workplace factors associated with burnout of family physicians. JAMA Intern Med 2017; 177 (07) 1036-1038
• 33 Robertson SL, Robinson MD, Reid A. Electronic health record effects on work-life balance and burnout within the I³ population collaborative. J Grad Med Educ 2017; 9 (04) 479-484
• 34 Shah M, De Arrigunaga S, Forman LS, West M, Rowe SG, Mishuris RG. Cumulated time to chart closure: a novel electronic health record-derived metric associated with clinician burnout. JAMIA Open 2024; 7 (01) ooae009
• 35 Tawfik DS, Phibbs CS, Sexton JB. et al. Factors associated with provider burnout in the NICU. Pediatrics 2017; 139 (05) e20164134
• 36 Tawfik DS, Shanafelt TD, Dyrbye LN. et al. Personal and professional factors associated with work-life integration among US physicians. JAMA Netw Open 2021; 4 (05) e2111575
• 37 Tawfik DS, Sinha A, Bayati M. et al. Frustration with technology and its relation to emotional exhaustion among health care workers: cross-sectional observational study. J Med Internet Res 2021; 23 (07) e26817
• 38 Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019; 2019: 136-144
• 39 Kang C, Sarkar IN. Interventions to reduce electronic health record-related burnout: a systematic review. Appl Clin Inform 2024; 15 (01) 10-25
• 40 Livaudais M, Deng D, Frederick T, Grey-Theriot F, Kroth PJ. Perceived value of the electronic health record and its association with physician burnout. Appl Clin Inform 2022; 13 (04) 778-784
• 41 Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19 (06) 349-357
• 42 Trockel M, Bohman B, Lesure E. et al. A brief instrument to assess both burnout and professional fulfillment in physicians: reliability and validity, including correlation with self-reported medical errors, in a sample of resident and practicing physicians. Acad Psychiatry 2018; 42 (01) 11-24
• 43 Creswell JW, Creswell JD. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications; 2017
• 44 Campbell JL, Quincy C, Osserman J, Pedersen OK. Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol Methods Res 2013; 42 (03) 294-320
• 45 Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. SAGE Publications; 2013
• 46 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 47 Tai-Seale M, Dillon EC, Yang Y. et al. Physicians' well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood) 2019; 38 (07) 1073-1078
• 48 Smith L, Kirk W, Bennett MM, Youens K, Ramm J. From headache to handled: advanced in-basket management system in primary care clinics reduces provider workload burden and self-reported burnout. Appl Clin Inform 2024; 15 (05) 869-876
• 49 Koranne R, Williams ES, Poplau S. et al. Reducing burnout and enhancing work engagement among clinicians: the Minnesota experience. Health Care Manage Rev 2022; 47 (01) 49-57
• 50 Johnson S, Yañez C, Panzer J, Stiehl E. Integration of primary care teams within the examination room: a qualitative study of advanced team-based care. J Ambul Care Manage 2022; 45 (01) 63-72
• 51 Rodríguez-Fernández JM, Loeb JA, Hier DB. It's time to change our documentation philosophy: writing better neurology notes without the burnout. Front Digit Health 2022; 4: 1063141
• 52 Shaw JG, Winget M, Brown-Johnson C. et al. Primary care 2.0: a prospective evaluation of a novel model of advanced team care with expanded medical assistant support. Ann Fam Med 2021; 19 (05) 411-418
• 53 Sinsky CA, Bodenheimer T. Powering-up primary care teams: advanced team care with in-room support. Ann Fam Med 2019; 17 (04) 367-371
• 54 Hung DY, Harrison MI, Truong Q, Du X. Experiences of primary care physicians and staff following lean workflow redesign. BMC Health Serv Res 2018; 18 (01) 274
• 55 Smith CD, Balatbat C, Corbridge S. et al. Implementing optimal team-based care to reduce clinician burnout. NAM Perspectives. National Academy of Medicine; 2018 https://nam.edu/implementing-optimal-team-based-care-to-reduce-clinician-burnout
• 56 Brown-Johnson CG, Chan GK, Winget M. et al. Primary care 2.0: design of a transformational team-based practice model to meet the quadruple aim. Am J Med Qual 2019; 34 (04) 339-347
• 57 Misurac J, Knake LA, Blum JM. The effect of ambient artificial intelligence notes on provider burnout. Appl Clin Inform 2025; 16 (02) 252-258
• 58 Linzer M, Poplau S, Prasad K. et al; Healthy Work Place Investigators. Characteristics of health care organizations associated with clinician trust: results from the healthy work place study. JAMA Netw Open 2019; 2 (06) e196201-e196201
• 59 Edwards ST, Marino M, Solberg LI. et al. Cultural and structural features of zero-burnout primary care practices. Health Aff (Millwood) 2021; 40 (06) 928-936
• 60 Linzer M, Sinsky CA, Poplau S, Brown R, Williams E. Healthy Work Place Investigators. Joy in medical practice: clinician satisfaction in the healthy work place trial. Health Aff (Millwood) 2017; 36 (10) 1808-1814

Publication History

Received: 11 December 2024

Accepted: 21 April 2025

Accepted Manuscript online:
28 April 2025

Article published online:
05 September 2025