Use of a Medication Safety Audit and Feedback Tool in the Emergency Department Is Affected by Prescribing Characteristics

Abstract

Background The Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department (EQUIPPED) program developed an audit and feedback health information technology (IT) solution with the intent to replace the in-person academic detailing service provided by the program. The EQUIPPED dashboard provides emergency department (ED) providers with a personalized view of their prescribing performance.

Objectives Here, we analyze the association between ED provider characteristics and viewership of the EQUIPPED dashboard, adding insight into strategies for addressing barriers to initial use.

Methods We performed a retrospective analysis of EQUIPPED dashboard viewership among four Veterans Affairs (VA) EDs. We extracted quantitative data from user interaction logs to determine evidence of dashboard use. Provider characteristics and baseline potentially inappropriate medication (PIM) prescribing rate were extracted from the VA's Corporate Data Warehouse. Logistic regression was used to examine the association between dashboard use and provider characteristics.

Results A total of 82 providers were invited to receive audit and feedback via the EQUIPPED dashboard. Among invited providers, 40 (48.7%) had evidence of at least 1 dashboard view during the 1-year feedback period. Adjusted analyses suggest that providers with a higher baseline PIM prescribing rate were more likely to use the dashboard (odds ratio [OR]: 1.22; 95% confidence interval [CI]: 1.01–1.47). Furthermore, providers at ED site D were more likely to use the dashboard in comparison to the other sites (OR: 9.99; 95% CI: 1.72–58.04) and reportedly had the highest site-level baseline PIM rate.

Conclusion Providers with lower PIM prescribing rates (i.e., <5%) receive communication from an integrated dashboard reminder system that they are “optimal prescribers” which may have discouraged initial attempts to view the dashboard. Site D had the highest baseline PIM rate, but further qualitative investigation is warranted to better understand why site D had the greatest users of the dashboard.

Keywords

medication management - emergency medicine - dashboard - incentives/barriers - diffusion of innovation

Background and Significance

With the emergence of the electronic health record (EHR) and enterprise data warehouses (EDWs), dashboards have become increasingly common within the health care setting.[1] Broadly speaking, a dashboard is a business intelligence tool that leverages data visualization techniques to provide actionable feedback that ideally leads to improved performance.[2] Within the health care setting, dashboards that display performance at the individual level are commonly referred to as clinical dashboards, while quality improvement dashboards aggregate performance data at the institution, ward, or clinic level to promote overall service improvement.[3] [4] More recently, clinical dashboards are being incorporated into day-to-day patient care in support of decision-making, medication safety, resource allocation, and to facilitate improved triaging of patients during outbreaks of infectious disease.[5] Perhaps the most appealing characteristic of a dashboard is its ability to synthesize vast amounts of data into a visually appealing format that results in improved data comprehension and reduced cognitive load.[6]

Clinical dashboards supported by a robust backend data architecture and an automated extract, transform, and load process can provide individual-level audit and feedback in near real time. The literature has thoroughly described the positive impact of audit and feedback at the individual level on clinical practice and health outcomes.[7] [8] However, when audit and feedback is provided solely through a clinical dashboard, evidence for positive impact on health outcomes remains inconsistent.[9] [10] [11] Poor dashboard utilization and lack of engagement may be responsible for the inconsistencies observed in the evidence base.[12] The integration of human factors principles during dashboard development followed by implementation of formal usability study can lead to improved dashboard satisfaction,[13] [14] [15] [16] [17] but beyond age,[18] there is little evidence on what factors may initially drive dashboard engagement once made available for routine use. Developing a better understanding of the facilitators and barriers to clinical dashboard utilization in different clinical settings would add further insight into dashboard orientation strategies and work practice implementation.[19]

Objectives

The Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department (EQUIPPED) program developed an audit and feedback health information technology (IT) solution (i.e., clinical dashboard) with the intent to replace the traditional in-person academic detailing service provided by the program. The EQUIPPED dashboard provides emergency department (ED) providers with a personalized view of their prescribing performance informed by the 2019 American Geriatrics Society (AGS) Beers Criteria for potentially inappropriate medications (PIMs) that should be avoided in patients of 65 years or older. [Fig. 1] contains a screenshot of the EQUIPPED dashboard landing page and highlights the dashboard's core audit and feedback components, which includes key performance indicators, peer-to-peer benchmarking, patient and encounter drill down, alternative medications, and longitudinal performance tracking. The data displayed in the EQUIPPED dashboard are sourced from the Veterans Affairs (VA) EHR and are refreshed nightly. The EQUIPPED dashboard was intended to be used passively rather than at point of care, serving as a continuous quality improvement information display. Further details of the EQUIPPED program, how the dashboard was developed, and the intended use of the dashboard can be found elsewhere.[20] [21] Here, we present an analysis of the association between ED provider characteristics and viewership of the EQUIPPED dashboard, adding insight into strategies for addressing barriers to use.

Fig. 1 The EQUIPPED dashboard landing page, which contains the following core audit and feedback components: key performance indicators (A), peer-to-peer benchmarking (B), individual patient or encounter drill down (C), alternative medications (D), and longitudinal performance tracking (E).[16]

Methods

Study Design and Data Source

We performed a retrospective analysis of EQUIPPED dashboard viewership among four VA EDs participating in the EQUIPPED program implementation trial that compared the impact of passive feedback provided through the dashboard to traditional in-person academic detailing on reducing PIM rates.[20] Dashboard viewership was measured by extracting quantitative data from dashboard user interaction logs. Provider characteristics and baseline PIM prescribing rates (i.e., prior to dashboard introduction) were extracted from the VA's Corporate Data Warehouse (CDW). The CDW is a national EDW comprised of clinical and administrative systems data.[22] At point of care, data are entered into the VA's EHR across all VA sites of care by way of manual entry, barcode scanning, or through automated instrumentation. The data from each EHR instance are then uploaded into the CDW on a nightly basis and organized into the relational format by logical domains. This analysis was intended to provide additional insight that will inform strategy on the delivery of an operations dashboard to VA clinician end users in the ED. Thus, the Salt Lake City VA Research and Development committee has classified this work as a nonresearch operational activity designed for internal VA purposes and determined that approval to develop and monitor the use of the dashboard by an Institutional Review Board (IRB) was not necessary. Access to the CDW for operational purposes is granted by the National Data Systems group, a division under the VHA Office of Informatics and Information Governance. Of note, the development and usage monitoring of the EQUIPPED dashboard is a separate activity from the comparative analyses done as part of the implementation trial, which was approved by the Emory and Durham VA Health Care System IRBs.

Study Population

During the implementation trial, providers at four VA ED sites had access to the dashboard for a 12-month period. Each VA ED had a designated EQUIPPED site champion that was oriented to the dashboard by the EQUIPPED team through virtual modalities, with the expectation of disseminating dashboard training materials (i.e., demonstration video and screenshots) to the ED providers at their local site. The EQUIPPED site champions did not have enhanced access to the dashboard to monitor ED providers at their sites. In summary, site champions did not engage in the audit and feedback process beyond providing a brief introduction to the EQUIPPED dashboard to their providers. Prior to issuing dashboard access, a list of ED providers with evidence of at least one discharge prescription written in the prior year was provided to each site champion at the four ED sites. The site champion then determined from this list who was still actively employed at their site and should be invited to receive prescribing feedback through the dashboard. The dashboard development team then sent a link to each ED provider that directed them to a Microsoft SharePoint site housed within the VA firewall where the dashboard could be viewed. The EQUIPPED dashboard is also supported by an integrated email system that communicates to the ED provider each month their prescribing performance and prompts them to access the dashboard. Thus, we anticipated that providers willing to engage with the dashboard would view it monthly at a minimum. Due to limited resources, we were unable to dynamically add new ED providers to the list of intended end users after dashboard feedback was initiated at each site. The start of dashboard access served as the index date for this analysis. The 12 months that followed the index date was the time window in which we determined who had viewed the dashboard.

Outcome

The primary outcome for this analysis was evidence of dashboard use (i.e., dashboard users vs. nonusers among those invited to receive feedback via the dashboard) structured as a dichotomous-dependent variable. Evidence of dashboard use was determined by identifying the number of unique ED providers that had at least one attempt to launch the dashboard during the 12-month feedback window that followed the index date (i.e., start of dashboard access). This outcome was chosen to measure which ED providers attempted to initially engage with the EQUIPPED dashboard once made available to them. We also examined the number of dashboard views during the 12 months of feedback for each provider that attempted to engage with the dashboard. This secondary outcome was structured as a continuous variable. However, we chose to only include the analysis of this secondary outcome as supplementary material (see [Supplementary Appendix A1], available in the online version). The statistical associations observed did not alter or further explain our main findings. Furthermore, we felt this particular outcome was better suited for seeking to understand the relationship between provider characteristics and sustained dashboard activity, rather than serving as a measure to understand drivers for initial dashboard viewership (i.e., work practice implementation).

Provider Characteristics

The provider characteristics included in this analysis are as follows: gender, age, duty basis, provider type, ED site (e.g., sites A, B, C, and D), and baseline PIM prescribing rate. Duty basis characterizes the provider's position as full time, part time, or flexible. Provider type identifies the provider as a physician or advanced practice provider (i.e., nurse practitioner and physician assistant). All provider characteristics except the baseline PIM prescribing rate were measured at the start of EQUIPPED dashboard access. The baseline PIM prescribing rate is a marker of prescribing performance prior to the ED provider being introduced to the EQUIPPED dashboard (i.e., starting audit and feedback). This rate was calculated by dividing the number of prescribed medications classified as category 1 PIMs according to the 2019 AGS Beers Criteria for older adults[23] (i.e., numerator) by the total number of medications prescribed at the time of ED discharge during a 1-year baseline period that preceded the index date (i.e., denominator). Only medications for patients 65 years of age or older at the time of dispensing were included. PIMs found within the 2019 AGS Beers Criteria with conditional recommendations (e.g., nitrofurantoin; avoid in individuals with a creatinine clearance >30 mL/min) were not classified as potentially inappropriate in the baseline PIM prescribing rate. Further, PIMs on the 2019 AGS Beers criteria that often require long-term monitoring and are primarily prescribed in a primary care setting were also not flagged as potentially inappropriate (e.g., use of proton pump inhibitors for >90 days). In summary, we followed the same marker for prescribing quality that the EQUIPPED program has previously used for program evaluation activities.[20]

Statistical Analysis

Proportions, means, and 95% confidence intervals (CIs) were calculated to compare provider characteristics between dashboard users versus nonusers. Unadjusted differences were analyzed using two independent sample t-tests for continuous variables and chi-square or Fisher's exact tests for categorical variables. Logistic regression was used to examine the relationship between dashboard use and provider characteristics, while controlling for potential confounding. The presented statistical model was mutually adjusted for all independent variables (i.e., gender, age, duty basis, provider type, ED site, and baseline PIM rate). Adjusted results are presented using odds ratios and 95% CIs. Negative binomial regression was used to examine the relationship between dashboard view count (i.e., secondary outcome), and provider characteristics (see [Supplementary Appendix A1], available in the online version). SAS version 9.4 (SAS Institute; Cary, NC) was used in performing all statistical analyses.

Results

A total of 82 providers among four ED sites were invited to access the EQUIPPED dashboard and review their prescribing performance. Among these targeted providers, 40 (48.7%) had evidence of at least one attempt to launch the dashboard, with a mean dashboard view count of 8.5 views during the 12-month feedback period. Unadjusted comparisons suggest that the baseline PIM rate and the ED site significantly differed between dashboard users versus nonusers (see [Table 1]). Dashboard users reportedly had a higher baseline PIM rate in comparison to nonusers (mean 8.01 vs. 5.58, p = 0.010). Notably, the highest proportion of dashboard users originated from site D (32.5%), even though site D represented the smallest number of ED providers (i.e., 15 providers) among all sites. Site D also reportedly had the highest site-level baseline PIM rate (mean 8.80; see [Table 2]). Adjusted analyses further suggest that providers with a higher baseline PIM prescribing rate were more likely to use the dashboard (OR: 1.22; 95% CI: 1.01–1.47) and originate from site D (OR: 9.99; 95% CI: 1.72–58.04; see [Table 3]). Gender, age, duty basis, and provider type were not found to be associated with dashboard viewership.

Table 1 Differences in provider characteristics between dashboard users vs. non-users among providers invited to receive feedback via the EQUIPPED dashboard

	All providers (N = 82)		Nonusers (N = 42)		Users (N = 40)		p-Value
	%/Mean	95% CI	%/Mean	95% CI	%/Mean	95% CI
Gender							0.812
Female	46.34	(35.56–57.13)	47.62	(32.51–62.72)	45.00	(29.58–60.42)
Male	53.66	(42.87–64.45)	52.38	(37.28–67.49)	55.00	(39.58–70.42)
Age (Mean)	50.50	(48.35–52.64)	49.35	(46.39–52.33)	51.70	(48.51–54.89)	0.280
Duty basis							0.868
Full time	75.61	(66.31–84.90)	76.19	(63.31–89.07)	75.00	(61.68–88.42)
Part time	18.29	(9.92–26.66)	19.05	(7.17–30.92)	17.50	(5.72–29.28)
Flexible	6.10	(0.92–11.28)	4.76	(0.00–11.20)	7.50	(0.00–15.66)
Provider type							0.589
Physician	81.71	(73.34–90.08)	80.90	(69.08–92.83)	82.50	(60.01–70.72)
APP	13.41	(6.04–20.79)	11.90	(2.11–21.70)	15.00	(3.93–26.07)
Missing	4.88	(0.22–9.54)	7.14	(0.00–14.93)	2.50	(0.00–7.34)
ED site							0.007
Site A	23.17	(14.04–32.30)	23.81	(10.93–36.69)	22.50	(9.56–35.44)
Site B	37.80	(27.31–48.30)	50.00	(34.88–65.12)	25.00	(11.58–38.42)
Site C	20.73	(11.96–29.51)	21.43	(9.02–33.84)	20.00	(7.60–32.40)
Site D	18.29	(9.92–26.66)	4.76	(0.00–16.16)	32.50	(17.99–47.01)
Baseline PIM rate (Mean)	6.76	(5.81–7.72)	5.58	(4.80–6.36)	8.01	(6.28–9.74)	0.010

Abbreviations: APP, advanced practice provider; CI, confidence interval; ED, emergency department; EQUIPPED, Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department; PIM, potentially inappropriate medication.

Table 2 Baseline PIM rates by site among the 82 emergency department providers invited to receive feedback via the EQUIPPED dashboard

ED Site	Mean	95% CI
Site A (19 providers)	7.79	(4.58–11.00)
Site B (31 providers)	5.65	(4.83–6.48)
Site C (17 providers)	5.85	(4.05–7.65)
Site D (15 providers)	8.80	(6.48–11.12)

Abbreviation: CI, confidence interval; ED, emergency department; EQUIPPED, Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department.

Table 3 The association between provider characteristics and dashboard users among the 82 Emergency Department Providers Invited to Receive Feedback via the EQUIPPED Dashboard

	Adjusted OR[a]	95% CI	p-Value
Gender
Female	0.79	(0.24–2.64)	0.701
Male (ref)	1.00	−	–
Age	1.01	(0.95–1.07)	0.791
Duty basis
Full time	1.75	(0.42–7.41)	0.445
Part time (ref.)	1.00	−	–
Flexible	6.23	(0.44–88.01)	0.176
Provider type
Physician (ref.)	1.00	–	–
APP	0.58	(0.09–3.66)	0.562
Missing	1.10	(0.08–14.58)	0.940
ED Site
Site A	1.08	(0.25–4.70)	0.919
Site B (ref.)	1.00	–	–
Site C	1.62	(0.41–6.35)	0.488
Site D	9.99	(1.72–58.04)	0.010
Baseline PIM rate	1.22	(1.01–1.47)	0.039

Abbreviations: APP, advanced practice provider; CI, confidence interval; ED, emergency department; EQUIPPED, Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department; OR, odds ratio; PIM, potentially inappropriate medication.

^a Adjusted on gender, age, duty basis, provider type, enrollment site, baseline PIM rate

Discussion

We hypothesized that dashboard viewership would be highest among younger ED providers as the literature has identified clear generational differences in the use of technology among clinicians.[18] [24] However, we were unable to observe differences in age between dashboard users and nonusers. In fact, ED providers with higher baseline PIM prescribing rates were more likely to engage with the EQUIPPED dashboard and baseline PIM prescribing rates appear to be more strongly associated with the initial attempt to view the EQUIPPED dashboard than any of the provider demographics we reviewed. A similar finding has been reported in an evaluation of the Salford Medication safety dASHboard (SMASH).[12] Developed for primary care teams in Salford (a city in Greater Manchester, United Kingdom), the SMASH dashboard[25] identifies patients that have been recently prescribed a high-risk medication based on prescribing safety indicators for general practitioners identified by Spencer et al.[26] Authors reported that SMASH Dashboard use was elevated among primary care practices with the greatest number of patients being prescribed high-risk medications.

The EQUIPPED program communicates to sites that a 30-day PIM rate of less than 5% is considered the threshold for “optimal prescribing.” Therefore, ED providers with a PIM rate greater than 5% may have felt a greater urgency to use the EQUIPPED dashboard and improve their prescribing behavior, while those near the 5% threshold may have felt less motivated to engage with the EQUIPPED dashboard. ED providers invited to receive feedback through the EQUIPPED dashboard that is at or below the 5% threshold are informed that they have achieved “optimal prescribing” through an automated email message delivered monthly. While providers above that PIM rate threshold are simply encouraged to access the dashboard and review their prescribing performance. Thus, the integrated email reminder language for “optimal prescribers” may have further discouraged a provider from initially attempting to view the dashboard if their 30-day PIM rate was at or below 5% at baseline. Adjusting how the optimal prescribing threshold is communicated through the dashboard interface and integrated email system should be considered so that providers classified as near-optimal prescribers remain motivated to use the dashboard and potentially improve their prescribing performance beyond the 5% threshold. For example, instead of communicating to the provider that they have achieved optimal prescribing, the feedback generated through the integrated email system could communicate whether the provider's 30-day PIM rate is trending up or down, or how their prescribing performance compares to their peers, with the invitation to review their 30-day PIM rate on the dashboard. The feedback intervention theory, which was used in developing the EQUIPPED dashboard and integrated email system, stresses the importance of communicating to providers when they have achieved or met applicable goals.[3] Thus, we are not suggesting that element of feedback be removed but instead are suggesting such information be accessible solely through viewing the dashboard to promote engagement.

We observed clear differences in dashboard viewership by the site. The highest proportion of dashboard users originated from site D. Notably, site D reportedly had the highest baseline PIM rate, which our analysis suggests plays a role in initial dashboard engagement. Additionally, site D was also the smallest site, comprised of 15 ED providers that received access to the EQUIPPED dashboard. Further, the largest ED site in our analysis (i.e., site B; 31 providers) reportedly engaged with the EQUIPPED the least, although this finding may be partly explained by the site's low baseline PIM rate. It is possible that smaller ED sites may be better candidates to receive prescribing feedback through the EQUIPPED dashboard due to a greater presence of the site champion among the ED provider group. Although site champions were not asked to engage in the audit and feedback process, we did not attempt to prevent the site champion from promoting the use of the dashboard to their ED providers. Thus, beyond ED size, the increase in dashboard engagement observed at site D may be partly explained by strategies implemented by the site champion to further market or promote dashboard engagement. Additionally, the literature has reported poor organizational adoption of medication safety tools when there is disagreement within the organization of the perceived value of the tool.[27] It is possible that ED providers at sites with less dashboard viewership were more likely to oppose the guidance found within the 2019 AGS Beers Criteria and responded by choosing not to use the dashboard. Further qualitative investigation is warranted to better understand the drivers of dashboard engagement not measured in this analysis. The literature reinforces the importance of involving “champions” in the work practice implementation of clinical dashboards to increase uptake and incentivize use.[28] Increasing the involvement of the site champion in the audit and feedback process among ED sites invited to use the dashboard is a potential modification of our implementation strategy that is currently under consideration. Larger ED sites may require multiple site champions to improve work practice implementation and achieve superior dashboard viewership. The literature reinforces this idea and suggests that a single-site champion may not be adequate when implementing new technology into work practices while also seeking to change provider behaviors.[29] Often multiple champions are needed for larger organizations and, ideally, each champion should also hold different organizational positions (i.e., managerial champion and front-line clinician champion) for optimal implementation.[29]

Limitations

Our results indicate that the baseline PIM rate is modestly associated with the viewership of the EQUIPPED dashboard. Additionally, we also have hypothesized that site D may have been more likely to engage with the EQUIPPED dashboard because the site champion could better manage the smaller provider group or perhaps more readily reinforced the dashboard's intended purpose and promoted its utility in comparison to the other site champions. Unfortunately, we were unable to statistically probe further the nature of the significant effects we observed due to our limited sample size. For example, we were interested in examining the potential synergistic effects (i.e., interaction term) between both baseline PIM rate and ED site but were unsuccessful as effect estimates became too unstable due to inadequate observations available for analysis for each ED site. Another limitation is that we did not track and measure ED provider turnover during the 12-month feedback period. It is possible that the larger ED sites had greater staffing turnover, which may have been exacerbated by the COVID-19 pandemic and could explain poorer dashboard engagement. In short, the email reminder system could have failed to reach ED providers that were no longer employed at the site. The findings from this analysis could clearly be strengthened through the inclusion of qualitative data that would allow us to gain additional perspective. For example, qualitative data could potentially reveal what strategies, if any, were used by each ED site champion to incentivize EQUIPPED dashboard use. Further review is warranted in qualitatively measuring site champion engagement with their providers and in understanding strategies implemented to promote dashboard use at each ED site.

Conclusion

Provider demographics, such as age, gender, and provider type may not represent the greatest barriers to clinical dashboard engagement. However, we did find that baseline prescribing performance was associated with dashboard viewership. Providers with an elevated baseline PIM rate were more likely to view the dashboard. EQUIPPED dashboard engagement may improve if feedback language is altered in the email reminder system to better promote dashboard viewership among providers already classified as “optimal prescribers” at the start of feedback. Beyond site D having an elevated baseline PIM rate, we are unable to further conclude why this site was also observed to have the greatest dashboard engagement. The lack of associations observed in this analysis further stresses the importance for conducting a comprehensive implementation evaluation using qualitative data techniques. This type of evaluation may provider further insight as to why ED site D was associated with higher dashboard viewership. Such an investigation may identify additional drivers of engagement beyond prescribing performance, such as site culture, leadership style of the site champion, and other unique strategies used at the site to promote dashboard use that are not measurable from the EHR. This additional context could be used to enhance work practice implementation of the EQUIPPED dashboard at future sites.

Clinical Relevance Statement

Clinical dashboards are unable to have an impact on clinical practice and health outcomes if utilization is poor. Monitoring the use of audit and feedback health IT solutions and the characterization of users provides insight into meaningful strategies for improving viewership.

Multiple-Choice Questions

1. Suppose you developed an audit and feedback clinical dashboard that identifies patients at risk for urinary incontinence and in need of screening. This dashboard contains an integrated email reminder system that delivers to the provider each month their urinary incontinence screening performance, expressed as a proportion (i.e., numerator = patients screened; denominator = patients seen at risk). The email reminder also contains a list of patients at risk for urinary incontinence with upcoming appointments for the current month and some language that encourages the clinician end user to view the dashboard for further details. How might you alter the feedback provided in the dashboard's integrated email system to promote increased use of the dashboard itself?

   • Reduce the amount of detail provided in the email by simply communicating to the provider the direction their screening performance is trending and encourage them to view the dashboard to identify actionable patients in need of screening with upcoming scheduled appointments.

   • Add more detail to the email, including additional data metrics found in the dashboard, so that the provider develops a greater interest in the dashboard's ability to improve health care delivery and patient outcomes, which would likely improve dashboard use.

   • Remove the email reminder system completely as it is distracting to the provider and without the reminder and its contents, providers would view the dashboard more often to obtain the information they need.

   • Have the email reminder also be sent to the provider's supervisor to promote greater accountability and dashboard engagement.

   Correct Answer: The correct answer is option a. The e-mail reminder system should summarize the direction their performance is trending and reiterate the value of viewing the dashboard to promote its use. Having too much information in the email may discourage use. The providers may find viewing the dashboard unnecessary since they have all the information they need, or they may ignore the email due to the presentation of excessive information. Although having the email reminder also sent to the provider's supervisor may be an attractive option, doing so may not be effective if the feedback in the email is not first optimized.

2. Suppose you have developed a quality improvement dashboard designed to improve the pneumococcal vaccination series completion rate among patients over 65 years of age. The dashboard is intended to be used by primary care providers at four clinics and displays to them a list of patients in their panel and each patient's pneumococcal vaccination status. The dashboard has an integrated email reminder system that prompts providers to view the dashboard each month. You have noticed that very few of the primary care providers have attempted to view the dashboard since it has been made available for routine use. How might you improve dashboard viewership among the target group of primary care providers?

   • Increase the frequency of the emails that are sent to each provider from the reminder system from monthly to weekly.

   • Present in the email reminder each primary care provider's performance data so that each user can compare themselves to their peers at the four clinics, motivating those that are less performing to access the dashboard.

   • Assign a site champion at each clinic to promote the quality improvement dashboard, including a brief orientation that summarizes its intended use. Provide the site champion access to the dashboard so that they can also evaluate dashboard activity periodically and incentive use among those less engaged.

   • Have your development team email each intended end user that they have not attempted to view the dashboard and if they don't attempt to do so soon their access will be revoked.

   Correct Answer: The correct answer is option c. The literature suggests that site champions are an important component for work practice implementation of a clinical dashboard to achieve adequate uptake. Too frequent email reminders may frustrate busy providers and result in alert fatigue. Further, if the monthly reminder interval has not been successful at encouraging at least one dashboard view among each user, it is unlikely more frequent reminders would perform better. Attempting to implement unblinded peer-to-peer benchmarking through the email reminder system also poses a data security threat and may discourage the use of the dashboard by providers due to privacy concerns. Lastly, attempting to motivate use by the development team may have little to no impact, or come across as a threat, as developers in this scenario do not have face time with the providers, are not viewed as clinical leaders, or have the clinical background necessary to promote and encourage performance improvement.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This work was characterized as a nonresearch operational activity and not human subject research.

• References

• 1 Ghazisaeidi M, Safdari R, Torabi M, Mirzaee M, Farzi J, Goodini A. Development of performance dashboards in healthcare sector: key practical issues. Acta Inform Med 2015; 23 (05) 317-321
• 2 Helminski D, Kurlander JE, Renji AD. et al. Dashboards in health care settings: protocol for a scoping review. JMIR Res Protoc 2022; 11 (03) e34894
• 3 Dowding D, Merrill J, Russell D. Using feedback intervention theory to guide clinical dashboard design. AMIA Annu Symp Proc 2018; 2018: 395-403
• 4 Randell R, Alvarado N, McVey L. et al. Requirements for a quality dashboard: lessons from national clinical audits. AMIA Annu Symp Proc 2020; 2019: 735-744
• 5 Ibrahim H, Sorrell S, Nair SC, Al Romaithi A, Al Mazrouei S, Kamour A. Rapid development and utilization of a clinical intelligence dashboard for frontline clinicians to optimize critical resources during Covid-19. Acta Inform Med 2020; 28 (03) 209-213
• 6 Khairat SS, Dukkipati A, Lauria HA, Bice T, Travers D, Carson SS. The impact of visualization dashboards on quality of care and clinician satisfaction: integrative literature review. JMIR Human Factors 2018; 5 (02) e22
• 7 Ivers N, Jamtvedt G, Flottorp S. et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; (06) CD000259
• 8 Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009; 47 (03) 356-363
• 9 Dowding D, Randell R, Gardner P. et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015; 84 (02) 87-100
• 10 Xie CX, Chen Q, Hincapié CA, Hofstetter L, Maher CG, Machado GC. Effectiveness of clinical dashboards as audit and feedback or clinical decision support tools on medication use and test ordering: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2022; 29 (10) 1773-1785
• 11 Twohig PA, Rivington JR, Gunzler D, Daprano J, Margolius D. Clinician dashboard views and improvement in preventative health outcome measures: a retrospective analysis. BMC Health Serv Res 2019; 19 (01) 475
• 12 Jeffries M, Gude WT, Keers RN. et al. Understanding the utilisation of a novel interactive electronic medication safety dashboard in general practice: a mixed methods study. BMC Med Inform Decis Mak 2020; 20 (01) 69
• 13 Richter Lagha R, Burningham Z, Sauer BC. et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020; 11 (04) 528-534
• 14 Murphy DR, Savoy A, Satterly T, Sittig DF, Singh H. Dashboards for visual display of patient safety data: a systematic review. BMJ Health Care Inform 2021; 28 (01) e100437
• 15 Nanji KC, Garabedian PM, Shaikh SD. et al. Development of a perioperative medication-related clinical decision support tool to prevent medication errors: an analysis of user feedback. Appl Clin Inform 2021; 12 (05) 984-995
• 16 Yarahuan JKW, Lo HY, Bass L, Wright J, Hess LM. Design, usability, and acceptability of a needs-based, automated dashboard to provide individualized patient-care data to pediatric residents. Appl Clin Inform 2022; 13 (02) 380-390
• 17 Burningham Z, Lagha RR, Duford-Hutchinson B. et al. Developing the VA Geriatric Scholars Programs' clinical dashboards using the PDSA Framework for quality improvement. Appl Clin Inform 2022; 13 (04) 961-970
• 18 Nakagawa K, Yellowlees P. Inter-generational effects of technology: why millennial physicians may be less at risk for burnout than baby boomers. Curr Psychiatry Rep 2020; 22 (09) 45
• 19 Elshehaly M, Randell R, Brehmer M. et al. QualDash: adaptable generation of visualisation dashboards for healthcare quality improvement. IEEE Trans Vis Comput Graph 2021; 27 (02) 689-699
• 20 Burningham Z, Jackson GL, Kelleher J. et al. The Enhancing Quality of Prescribing Practices for Older Veterans Discharged From the Emergency Department (EQUIPPED) potentially inappropriate medication dashboard: a suitable alternative to the in-person academic detailing and standardized feedback reports of traditional EQUIPPED?. Clin Ther 2020; 42 (04) 573-582
• 21 Stevens M, Hastings SN, Markland AD. et al. Enhancing Quality of Provider Practices for Older Adults in the Emergency Department (EQUiPPED). J Am Geriatr Soc 2017; 65 (07) 1609-1614
• 22 Fihn SD, Francis J, Clancy C. et al. Insights from advanced analytics at the Veterans Health Administration. Health Aff (Millwood) 2014; 33 (07) 1203-1211
• 23 By the 2019 American Geriatrics Society Beers Criteria® Update Expert Panel. American Geriatrics Society 2019 Updated AGS Beers Criteria for potentially inappropriate medication use in older adults. J Am Geriatr Soc 2019; 67 (04) 674-694
• 24 Ngusie HS, Kassie SY, Chereka AA, Enyew EB. Healthcare providers' readiness for electronic health record adoption: a cross-sectional study during pre-implementation phase. BMC Health Serv Res 2022; 22 (01) 282
• 25 Williams R, Keers R, Gude WT. et al. SMASH! The Salford medication safety dashboard. J Innov Health Inform 2018; 25 (03) 183-193
• 26 Spencer R, Bell B, Avery AJ, Gookey G, Campbell SM. Royal College of General Practitioners. Identification of an updated set of prescribing—safety indicators for GPs. Br J Gen Pract 2014; 64 (621) e181-e190
• 27 Grant A, Dreischulte T, Guthrie B. Process evaluation of the Data-driven Quality Improvement in Primary Care (DQIP) trial: case study evaluation of adoption and maintenance of a complex intervention to reduce high-risk primary care prescribing. BMJ Open 2017; 7 (03) e015281
• 28 Waltz TJ, Powell BJ, Matthieu MM. et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci 2015; 10: 109
• 29 Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: An integrative review of champions in healthcare-related implementation. SAGE Open Med 2018; 6: 2050312118773261

Publication History

Received: 21 February 2023

Accepted: 17 May 2023

Article published online:
30 August 2023