Developing the VA Geriatric Scholars Programs' Clinical Dashboards Using the PDSA Framework for Quality Improvement

Abstract

Background Involving clinician end users in the development process of clinical dashboards is important to ensure that user needs are adequately met prior to releasing the dashboard for use. The challenge with following this approach is that clinician end users can undergo periodic turnover, meaning, the clinicians that played a role in the initial development process may not be the same individuals that use the dashboard in future.

Objectives Here, we summarize our Plan, Do, Study, Act (PDSA)-guided clinical dashboard development process for the VA Geriatric Scholars Program (GSP) and the value of continuous, iterative development. We summarize dashboard adaptations that resulted from two PDSA cycles of improvement for the potentially inappropriate medication dashboard (PIMD), one of many Geriatric Scholars clinical dashboards. We also present the evaluative performance of the PIMD.

Methods Evaluation of the PIMD was performed using the system usability scale (SUS) and through review of user interaction logs. Routine end users that were Geriatric Scholars and had evidence of 5 or more dashboard views were invited to complete an electronic form that contained the 10-item SUS.

Results The proportion of Geriatric Scholars that utilized the PIMD increased for each iterative dashboard version that was produced as a byproduct from feedback (31.0% in 2017 to 60.2% in 2019). The overall usability of the PIMD among routine users was found to be above average (SUS score: 75.2 [95% CI 70.5–79.8]) in comparison to the recommended standard of acceptability (SUS score: 68)

Conclusion The solicitation of feedback during dashboard orientations led to iterative adaptations of the PIMD that broadened its intended use. The presented PDSA-guided process to clinical dashboard development for the VA GSP can serve as a valuable framework for development teams seeking to produce well-adopted and usable health information technology (IT) innovations.

Keywords

visualization - American Geriatrics Society - clinical dashboard - early-stage usability - dashboard - electronic health record - quality improvement

Background and Significance

With the advent of the electronic health record (EHR), institutions can amass a trove of information about patient health, provider behaviors, team behaviors, institutional processes, and patient outcomes.[1] Initially, health care organizations attempted to synthesize this information into structured reports to monitor and track important quality indicators of care. Unfortunately, these reports were static in nature and lacked interactivity, which did not lead to more effective decision making and care delivery.[2] Due to this limitation, near-real time dashboards soon became mainstream across the health care sector for clinical quality performance measurement and monitoring. Such dashboards are capable of synthesizing and packaging EHR data into a visually appealing and user-friendly format, providing continuously updated information with point-and-click data filtering interactivity in effort to stimulate change.[3]

Distinctions are well defined in the literature between quality improvement (QI) dashboards and clinical dashboards. QI dashboards seek to measure and monitor the performance of health care delivery at the facility or organizational level to inform operational decision-making.[4] These systems reinforce a “top down,” or institution-initiated process for safety and QI. On the contrary, clinical dashboards are designed to provide audit and feedback on clinical performance to individual providers based on set standards or in comparison to their peers.[5] [6] However, clinical dashboards still possess features that lead to enhanced decision making in support of local or microlevel QI.

Standalone evaluative studies of clinical dashboards are becoming more mainstream. These studies report on dashboard performance as measured by end user uptake, usability, or associated improvement in health outcomes, with some detail provided on which factors in their development process were believed to contribute to the dashboard's successes.[7] [8] [9] [10] [11] [12] There is growing consensus that involving clinicians in the development process of audit and feedback clinical dashboards is important to ensure that user needs are adequately met prior to releasing the dashboard for routine use, subsequently improving uptake and adoption.[13] [14] This principle has long been understood in the field of human-centered design, which seeks to systematically integrate end-user feedback throughout the development process.[15] Unfortunately, analytic and informatics teams have been slow to adopt human-centered design principles when developing clinical dashboards. A recent systematic review comprised of 33 clinical dashboard evaluation studies reported that only four had leveraged human factors principles during development.[16] The challenge with following a human-centered design approach becomes apparent in cases where the target group of clinician end users experiences periodic turnover, meaning, the clinicians that played a role in the initial development process of the dashboard may not be the same individuals that use the dashboard in the future.[17] Thus, it emphasizes the need for a development framework that contains a continual feedback mechanism, even after the dashboard is made available for routine use. Similarly, we believe there is an important parallel that is often overlooked between clinical dashboard development and the software development life-cycle, which originates from the field of computer science.[18] That is, the software development process is dynamic, cyclical, and never ending. Rarely do you ever a see a “finished” software product. Thus, we believe that a clinical dashboard should rarely be declared “finished” once made available for routine use. Unfortunately, this parallel is rarely recognized or accommodated in the evaluative literature of clinical dashboards.

Objectives

Here, we summarize our clinical dashboard development process and demonstrate the value of a continual solicitation feedback mechanism for the Veterans Affairs (VA) Geriatric Scholars Program (GSP), a national workforce development initiative that aims to improve the quality of health care that older Veterans receive in VA interdisciplinary primary care settings. The GSP program comprises of a combination of intensive didactics, a day-long interactive QI workshop, and the application of knowledge through completion of a local QI project. The program follows an annual cycle, enrolling new primary care providers (e.g., physician, physician assistant, advanced practice nurse) and primary care support team members (e.g., psychologists, pharmacists, social workers, rehabilitation therapists) each year. Further details of the program can be found elsewhere.[19] A suite of clinical dashboards have been developed for primary care providers and associated health professionals that participate in the GSP. These dashboards facilitate the completion of micro-QI projects at their local primary care clinics by reducing the burden of collecting baseline data on primary care patient panels and allow for seamless tracking of clinical performance using visualization of process and outcomes measures. We describe in detail our clinical dashboard development process that was guided by the Plan, Do, Study, Act (PDSA) framework for QI. Furthermore, we demonstrate the value of this development process by presenting specific adaptations that resulted from two PDSA cycles of improvement over a 3-year period for the Potentially Inappropriate Medication Dashboard (PIMD); one of many Geriatric Scholars clinical dashboards actively in use. We also present the evaluative performance of the PIMD over the same period to demonstrate the effectiveness of our development process.

Methods

Dashboard Development Process

The GSP clinical dashboard development process is comprised of five steps: (1) identify target geriatric-focused clinical practice guideline; (2) identify subject matter expert (SME), (3) develop prototype; (4) implement early-stage usability testing in a controlled setting; and (5) orient end users, solicit feedback, and release dashboard for routine use. [Fig. 1] summarizes this development process and maps the process to the Plan, Do, Study, Act (PDSA) framework. The PDSA framework is a widely used strategy for QI that supports continuous learning through testing of changes.[20] [21] The PDSA framework consists of four phases for testing changes: (1) plan to test a proposed improvement; (2) do, or execute the plan; (3) study the results; and (4) act on these results. This cycle repeats continuously, creating a mechanism for careful study of ongoing modifications and improvements. We relied on the PDSA framework to guide our dashboard development process due to the emphasis it places on iterative adaptations. Additionally, the dashboard development team is also comprised of QI methodologists making the framework an ideal foundation for development due to its familiarity. The Agile software development process is arguably a framework of close equivalence that is recognized in field of computer science.[22] Unique to the PDSA framework is the importance of mechanisms being put in place for continuous, ongoing evaluation to improve system design and functionality. Thus, using the PDSA framework as our guide, clinical dashboard development for the GSP continues well after the initial production-ready version is made available for routine use.

Fig. 1 PDSA-guided dashboard development process. PDSA, plan, do, study, act.

Step 1: Identify Target Geriatric-Focused Clinical Practice Guideline or Quality Indicator

Each of the clinical dashboards developed for the GSP are themed and can be differentiated by the core QI performance measures they report. These QI performance measures were based on geriatric-focused clinical practice guidelines or quality indicators and reflect topics of QI projects that were commonly implemented by the Geriatric Scholars prior to our involvement in the program.[23] The current suite of GSP clinical dashboards actively in use in the VA identify patients at risk for pneumococcal disease, geriatric psychiatric admissions, ambulatory care sensitive condition (ACSC) hospitalizations, osteoporosis, and adverse drugs events due to inappropriate prescribing ([Fig. 2]). Currently, a fall risk assessment dashboard and a coronavirus disease 2019 (COVID-19) vaccination dashboard are under development.

Fig. 2 Screenshot of the geriatric scholars program's suite of clinical dashboards.

Step 2: Identify SME

Once a geriatric-focused topic was chosen to serve as the foundation by which a clinical dashboard would be built, we then identified an SME. In addition to expertise in the state-of-the-art in practice, the SME was required to be internal to the VA to assure that the product met current VA guidance and recommendations. The SME provided valuable insight into how care was documented in the VA's EHR as it pertained to the guideline. This insight aided the development team in determining feasibility of the dashboard build and provided needed direction on where supporting data could be found within the VA's Corporate Data Warehouse (CDW), a national data repository comprised of multiple administrative and clinical systems data. At point of care, data are entered into the VA's Veterans Health Information Systems and Technology Architecture (VistA) EHR system by way of manual entry, barcode scanning, or through automated instrumentation. These data are then uploaded into the VA's CDW on a nightly basis.

Step 3: Develop Dashboard Prototype

A participatory design approach was followed in defining core dashboard requirements and developing initial prototypes.[24] This approach fosters a sense of ownership and accountability across stakeholders, including the dashboard development team, GSP leadership, faculty that teach the interactive workshop on QI methodology, and the SME. The QI workshop faculty served as valuable sources of guidance in ensuring the clinical dashboards aligned with the QI workshop curriculum. Through this effort consensus was reached on the critical data elements that should be displayed (e.g., upcoming primary care appointment date), how performance measures should be visualized (e.g., longitudinal trends), appropriate vocabulary that should be used in the dashboard to represent key QI concepts, and the necessity for drilldown functionality so that actionable patients could be seamlessly identified in need of intervention. Iterative development with periodic internal review proceeded until the prototype was deemed ready for early-stage usability testing.

Step 4: Early-Stage Usability Testing in a Controlled Setting

During early-stage development, usability testing was conducted in a controlled setting among a small number of testers (n = 3). Testers were primary care providers or primary care team support clinicians (e.g., psychologists, pharmacists, social workers, and rehabilitation therapists) in the VA who were alumni of the GSP. This allowed us to engage with clinical end users early in the development process in support of human-centered design. Our usability approach was based on key elements found within “The Research-Based Web Design and Usability Guidelines,” developed by the U.S. Department of Health and Humans Services.[25] Methods of usability assessment included task analysis, direct observation, and retrospective probing. Further detail on the development of testing protocol, test administration strategies, usability tasks, and the resulting quantitative usability performance and emerging themes of qualitative feedback through cognitive interviewing can be found elsewhere.[26]

Step 5: Orient End Users, Solicit Feedback, Release Dashboard for Routine Use

In conjunction with the day-long QI workshop (continuing medical education 6.5 hours), Geriatric Scholars are oriented to the clinical dashboards and trained on how they can be used as they pursue implementation of a QI project at their local institution. The orientation involves live demonstration of each clinical dashboard, explanation of navigation best practices (i.e., drilldown), and an interactive discussion with the Geriatric Scholars on their views of potential limitations and barriers to using the dashboards for their QI projects. This method of dashboard delivery and solicitation of feedback through interactive discussion triggers additional PDSA development cycles of refinement and improvement. Feedback that is produced from the dashboard orientation is documented and then discussed during monthly meetings with the participatory team. Changes are prioritized based on feasibility and whether proposed needs are being expressed across multiple end users (i.e., emerging themes). Once changes are agreed upon, a PDSA cycle is triggered and steps 1 through 4 are repeated. If implemented feedback does not result in substantial changes to the overall design of the dashboard, small-scale usability (i.e., step 4) testing is not repeated. After feedback is implemented, the new version of the dashboard is not released into production for routine use until the next enrollment year. This ensures the version of the clinical dashboard of the Geriatric Scholar is oriented to remain consistent throughout the duration of their QI project.

The Potentially Inappropriate Medication Dashboard

We chose to use the PIMD to evaluate the performance of our clinical dashboard development process. Developed in 2016 and made available for routine use by the broader GSP in 2017, the PIMD was created to assist Geriatric Scholars with the implementation of a local QI project that aims to reduce the number of potentially inappropriate medications issued to older Veterans. Potentially inappropriate medications were identified from the 2019 American Geriatrics Society (AGS) Beers Medication Criteria.[27] The PIMD is comprised of a provider summary view, patient summary view, and patient detail view. The provider summary view is the dashboard landing page and contains aggregated data at the end of user's primary care panel level. This view contains the number of patients 65 years of age and older, average active medication count, and the number of patients in the panel with an active prescription for a potentially inappropriate medication. The patient summary view is a drill through report accessible from the provider summary view that displays the list of patients (i.e., name and last four digits of SSN) actively on a potentially inappropriate medication, including an Rx count if currently being prescribed more than one medication found on the 2019 Beers Medication Criteria. Lastly, the patient detail view is a drill through report accessible from the patient summary view and contains patient details (i.e., pertinent demographics and comorbidities) and the pharmacotherapy profile of the potentially inappropriate medication(s), including generic name, therapeutic class, issue date and dose, among other elements. Further details on the PIMD can be found elsewhere.[26]

PDSA Cycles

The PIMD underwent two PDSA cycles of improvement from 2017 to 2019. As a result, three versions of the PIMD were in use from 2017 through 2019. Each iteration of the PIMD replaced the previous version, with the newer version being made available for use at the start of the next enrollment year; multiple versions were not concurrently in use. PDSA cycle 1 was triggered after feedback was provided by Geriatric Scholars during the first dashboard orientation in 2017. Feedback was implemented and a new iteration of the PIMD (i.e., version 1.1) was deployed prior to the 2018 enrollment class being oriented. The new iteration consisted of modifying the available baseline prescribing performance measures found on the summary view landing page. As previously described, Version 1.0 of the PIMD displays the number of patients in a primary care panel actively on a potentially inappropriate medication ([Fig. 3]). This patient count and proportion (%) intended to serve as the core QI performance measure by which a Geriatric Scholar could develop a formal aim statement and use as a baseline to measure the impact of their QI efforts. However, Geriatric Scholars who were primary care providers voiced that this metric did not clearly represent their prescribing performance and instead represented a measure of health care quality that they felt was partially out of their control or management. These Geriatric Scholars stressed that many patients in their panels are actively on high-risk medications prescribed by other providers the Veteran had obtained care from (i.e., specialty care). To address this concern, we developed an additional measure of prescribing performance to display on the landing page of the PIMD, version 1.1; the number of potentially inappropriate medications prescribed by the primary care provider in the last 6 months ([Fig. 4]). This new proportion (%) only included medications prescribed by the primary care provider, who often is the Geriatric Scholar, and used a 6-month look back period which matches the expected duration of time they are allotted to complete their QI project.

Fig. 3 The PIM Dashboard Version 1.0 (provider summary view): displaying the number and proportion (%) of patients actively on an PIM for a given primary care patient panel. PIM, potentially inappropriate medication.

Fig. 4 The PIM Dashboard Version 1.1 (Provider Summary View): displaying an added measure of prescribing performance; the number and proportion (%) of PIMs prescribed only by the primary care provider in the last 6 months. PIM, potentially inappropriate medication.

PDSA cycle 2 was triggered after feedback was provided by Geriatric Scholars during the dashboard orientation in 2018. The adaptions that resulted from the feedback were deployed (i.e., version 1.2) prior to the 2019 enrollment class being oriented to the PIMD. During the 2018 dashboard orientation, several Geriatric Scholars expressed being overwhelmed with implementing a QI project that required monitoring nearly 100+ medications found on the 2019 AGS Beers Criteria list.[27] Instead, Geriatric Scholars wanted the option to be able to quickly identify a cohort of patients in a primary care panel actively being prescribed a certain high-risk medication that fell within a specific therapeutic class, such as proton pump inhibitors or benzodiazepines. This would allow the end user the ability to implement a QI project that focused on reducing the prescribing of a specific subset of high-risk medications that they felt were of greatest concern for their target population. As a result of this feedback, a nested group expansion feature was added in Version 1.2 of the PIMD. This feature allowed the end user to identify all patients in their panel actively on a potentially inappropriate medication and then further break that patient count down into meaningful groups based on therapeutic classes ([Fig. 5]). These new patient counts were also setup to function as filtered drill throughs to the patient summary view.

Fig. 5 PIM Dashboard Version 1.2 (provider summary view): displaying the nested group expansion feature that was added in Version 1.2 of the PIMD. This feature allowed the end user to identify patients in their panel actively on potentially inappropriate medications that fall within specific therapeutic classes. PIMD, potentially inappropriate medication dashboard.

Evaluation Design

Evaluation of the PIMD was performed by measuring dashboard utilization among Geriatric Scholars from 2017 to 2019. Dashboard utilization was not measured in 2020 as the GSP paused enrollment due to the COVID-19 pandemic. Dashboard utilization by non-Geriatric Scholars (i.e., program stakeholders, Geriatric Scholar clinic teams, and development team members) was not measured and included in this evaluation. To determine PIMD uptake, we extracted quantitative data from user interaction logs, which allowed us to determine a unique user count and dashboard view count. The dashboard view count reflects the number of times the landing page of the dashboard was viewed and does not take into consideration drill-down executions. We also quantified usability of the PIMD using the industry standard system usability scale (SUS). The 10-item SUS was integrated into an electronic form using InfoPath and published to SharePoint. All PIMD end users with more than five dashboard views during their class enrollment year (i.e., 2017–2019) were invited to complete the electronic form that contained the 10-item SUS. Responses were collected within a secured SharePoint list and then extracted for analysis. Proportions, means, standard deviations, and confidence intervals were calculated to summarize PIMD utilization and SUS performance.

Results

The proportion of Geriatric Scholars that utilized the PIMD (i.e., evidence of at least one attempt to view dashboard) increased for each iterative dashboard version that was produced (31.0% in 2017 to 60.2% in 2019; see [Fig. 6]). We identified 44 Geriatric Scholars with evidence of five or more dashboard views during their class enrollment year. The 10-item SUS was completed by 28 of these end users (response rate = 63.6%). The mean number of dashboard views among all end user responders was 66.4 (95% CI 25.2–102.8). The overall usability of the PIMD, irrespective of versioning, was found to be above average (SUS score: 75.2 [95% CI 70.5–79.8]) in comparison to the recommended standard of acceptability (SUS score: 68). Versions 1.1 (SUS score: 77.5 [95% CI 67.8–87.2]) and 1.2 (SUS score: 74.9 [95% CI 68.8–81.0] appeared to perform better than version 1.0 (SUS score: 71.7 [95% CI 62.6–80.8]), although differences were not statistically significance. Usability performance also appeared elevated among “frequent users” (≥20 views; SUS Score: 79.1 [95% CI 72.3–85.9]; see [Table 1]), irrespective of version, although most “frequent users” were users of version 1.2 (i.e., during final year of evaluation), which may be a byproduct of version 1.2 containing additional drill throughs which can lead to repeat navigation to the landing page during exploration.

Table 1 Quantification of PIM dashboard usability by version and utilization

	Number of responding end users[a]	Usability performance (Standard of acceptability = 68.0)
	N	SUS score	95% CI
Overall usability	28	75.2	70.5–79.8
Version 1.0[b]	3	71.7	62.6–80.8
Version 1.1[c]	7	77.5	67.8–87.2
Version 1.2[d]	18	74.9	68.8–81.0
Frequent users (≥20 views)	14	79.1	72.3–85.9
Infrequent users (<20 views)	14	71.3	65.5–77.0

Abbreviation: PDSA, Plan, Do, Study, Act.

^a Greater than five dashboard views.

^b No adaptions.

^c PDSA cycle 1.

^d PDSA cycle 2.

Fig. 6 Proportion (%) of Geriatric Scholars with evidence of PIM dashboard use by class enrollment year. PIM, potentially inappropriate medication.

Discussion

We have described in detail our clinical dashboard development process for the GSP and have pursued evaluation of the PIMD among real-world end users to assess the effectiveness of a PDSA guided development process with a continuous feedback mechanism. Ideally, we would have liked to have seen version 1.2 of the PIMD produce the best SUS score to strengthen our hypothesis that a dashboard which undergoes continuous development based on feedback from end users would result in a steady and consistent improvement in usability performance. Notably, the SUS estimates for versions 1.0 and 1.1 were less stable due to a fewer number of end users available for analysis, making it difficult to fully reject this hypothesis. Regardless, we were pleased to see that the PIMD was found to be usable among a large sample of routine end users. There are likely multiple contributing factors for this success, but there are two that we think are significant. First, our development process prioritizes the needs of the clinician end user, as demonstrated by integration of clinical SMEs to ensure our reporting is accurate and consistent with VA recommendations. Further, we involve clinician end users early in the build process by means of small-scale usability testing immediately after prototyping. Engaging with potential clinician end users early in the development process is a proven strategy that has been shown be successful in defining audit and feedback reporting requirements and proactively addressing unanticipated end user needs.[28] Second, we follow widely accepted design best practices informed by the feedback intervention theory (FIT).[6] [29] [30] The FIT aims to delineate how feedback should be organized and delivered to lead to behavior change and performance improvement. For example, timeliness of feedback is a known critical element that is needed to improve performance.[31] Each of the GSP's dashboards are supported by an automated extract, transform, and load (ETL) data process that executes daily to ensure that the information found within the dashboards are near-real time. For the PIMD, there is little to no delay between the prescribing of a potentially inappropriate medication and when that prescription is visible in the dashboard. Furthermore, the PIMD is organized into three data views or layers of drilldown (i.e., provider, patient, and detail). Drilldown functionality is critical to ensuring the end user can identify actionable patients in need of intervention, another notable best practice design of feedback interventions.[30]

We hypothesize that the observed increase in the proportion of Geriatric Scholars who used the PIMD, as new versions were released, has largely been due to our PDSA-guided framework for development, and in particular the dashboard orientation. Beyond serving as a mechanism for continual feedback, the dashboard orientation is a platform that allows us to demonstrate the value of dashboards and teaches potential users how to use the tools in support of their QI efforts. The literature has demonstrated the importance of dashboard delivery, including its significance on uptake, provider behavior change, and improved health outcomes.[6] Simply creating a dashboard and providing the navigation pathway or web address to the dashboard is often not sufficient in promoting uptake and adoption of use. We have found that formal delivery and orientation of a clinical dashboard to the end user identify additional barriers of use and communicate confidence to the end user that they will have the technical support needed to overcome challenges.

Notably, the iterative adaptations we have made to our dashboards from the feedback provided during the orientations has broadened their intended uses. For example, prior to PDSA cycle 2, interested end users of the PIMD included Geriatric Scholars largely passionate about the AGS Beers Criteria. After adding in functionality that supported the categorization of patients on a potentially inappropriate medication by therapeutic class, we found the general appeal of the PIMD to improve. Geriatric Scholars interested in reducing the prescribing of specific medications or subclasses of medications now had the ability to do so. This broadening of intended uses became evident after we began to see an increase in the variation of QI project topics being submitted by PIMD users.

A known strength of this work is the implementation of usability measurement among real-world end users, post-deployment of the PIMD. We have described the importance of engaging with clinician end users early in the clinical dashboard development process by means of small-scale usability study. However, quantification of usability of a production-ready clinical dashboard after it has been made available for routine use and among a larger sample of end users is also important to ascertain the true performance of the user experience and in evaluating the development framework.[32] The literature suggests that usability testing of health information technology (IT) tools and systems should be performed at different stages of the system development lifecycle.[33] Therefore, one should not think of usability assessment as a finite activity, but rather an iterative and continuous process. The timing of usability assessment determines what system components and characteristics can be evaluated. For example, early-stage usability testing can provide insight into user perception and satisfaction with the system, but only for a set of predefined tasks given to the tester. On the contrary, usability assessment at the stage of routine use can provide meaningful insight into efficiency and effectiveness for the system's ability to perform its intended use (e.g., QI). This leads us to a limitation of this work. We did not evaluate the impact of the PIMD on prescribing behavior, health outcomes, or rate of QI project completion among routine users. This is an important area of future work that would allow us to evaluate the FIT-guided design principles and our PDSA-guided development framework more fully. An additional limitation is that the dashboard view count extracted from our execution logs may not be fully reflective of real-world PIMD uptake and may overestimate utilization. There may have been circumstances where an end user had to repeatedly refresh the dashboard when encountering technical issues or in first becoming familiar with the dashboard, resulting in the counting of unintentional views. In addition to these limitations, we must also mention a notable challenge to implementing a development framework that is iterative, dynamic, and continuous. This approach requires continual funding streams to sustain development activities. To address this barrier, the participatory team can review the feasibility of proposed changes ascertained from feedback channels, then prioritize the modifications that fit within the bounds of available resources.

Conclusion

We have described our PDSA-guided clinical dashboard development process for the VA GSP and demonstrated its perceived value through measuring dashboard uptake and quantification of usability among routine users of the PIMD. Due to the COVID-19 pandemic, Geriatric Scholar enrollment was put on hold in 2020 and utilization of the PIMD did not occur during the year. In 2021, enrollment restarted, and we began drafting this manuscript soon after virtual dashboard orientations were completed in September of 2021. There were no significant feedbacks that triggered an additional PDSA cycle in 2021. As a result, PIMD version 1.2 remained active and in use throughout the year and into 2022. The PIMD is currently undergoing significant modifications as we migrate to a cloud-based reporting platform, further highlighting the importance of a development framework that supports continuous and iterative changes. In summary, the value of a continual feedback mechanism within a system's development lifecycle must not be overlooked. Given the dynamic nature of the health care system, the variability of systems in which clinicians operate, and the unknown challenges they may face, we must leave room for continued development to respect the needs of our end users and maintain relevance. Further, stakeholders should expect to invest long term into the development of clinical dashboards and not assume that development should cease at a specific point in time.

Clinical Relevance Statement

The presented PDSA-guided process to clinical dashboard development for the VA GSP can serve as a valuable framework for development teams and organizations seeking to produce well-adopted health IT tools and solutions.

Multiple Choice Questions

1. Suppose you and your development team are tasked with building a clinical dashboard that identifies actionable patients at risk for osteoporosis and in need of screening. During development, you ensure your clinical dashboard possesses important qualities described by the Feedback Intervention Theory to maximize the dashboard's ability to improve osteoporosis screening practices among clinician end users. Which dashboard quality is an example of a feedback design best practice informed by the Feedback Intervention Theory?

   • The dashboard is supported by an ETL that performs a daily refresh of the data.

   • The dashboard was designed using a color scheme that is colorblind friendly.

   • The dashboard is supported by a data definition document and user guide.

   • The dashboard supports the exportation of data to excel for further analysis.

   Correct Answer: The correct answer is option a. Each of the listed dashboard qualities are important to producing a usable product. However, ensuring that the data which supports the dashboard is near real-time will assist the end user in the identification of patients in immediate need of intervention and supports timely feedback to the clinician on their performance so that changes in clinical processes can be quickly made, when necessary.

2. Suppose version 1.0 of your osteoporosis risk assessment dashboard has been released for routine use for 2 years but has experienced poor uptake and adoption in recent months. You are aware that the initial end users included primary care providers, but the end user base has since shifted to also include registered nurse care managers. How might the integration of a continuous feedback mechanism within the interface of the dashboard or through other strategies of solicitation best increase the uptake and adoption of the osteoporosis risk assessment dashboard among all end users?

   • A continuous feedback mechanism can support the aggregation and organization of qualitative data, which if positive, can be used for marketing to other potential clinician end users.

   • A continuous feedback mechanism builds end user confidence in the dashboard system and communicates to the end user that there is available IT support.

   • A continuous feedback mechanism would not support improvement in the uptake and adoption of a clinical dashboard, such feedback is only valuable during early-stage development.

   • A continuous feedback mechanism can identify barriers to use and provide direction on strategies to broaden the intended use of the dashboard.

   Correct Answer: The correct answer option is d. Sharing of positive feedback obtained from the continuous feedback mechanism to other potential end users may generate initial interest in the dashboard but is unlikely to improve sustained uptake and utilization if barriers to use remain. Further, a continuous feedback mechanism may communicate to the end user that they will be adequately supported, but such confidence would unlikely overcome observable flaws in the dashboard. A continuous feedback mechanism or loop is often designed to identify areas in need of improvement, including barriers to use, which when addressed, can increase the uptake and adoption of the clinical dashboard by broadening appeal to a greater end user base.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This work was characterized as a non-research operational activity and not human subject research.

• References

• 1 Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform 2016; (Suppl. 01) S48-S61
• 2 Ghazisaeidi M, Safdari R, Torabi M, Mirzaee M, Farzi J, Goodini A. Development of performance dashboards in healthcare sector: key practical issues. Acta Inform Med 2015; 23 (05) 317-321
• 3 Lau MK, Bounthavong M, Kay CL, Harvey MA, Christopher MLD. Clinical dashboard development and use for academic detailing in the U.S. Department of Veterans Affairs. J Am Pharm Assoc (Wash DC) 2019; 59 (2S): S96
• 4 Randell R, Alvarado N, McVey L. et al. Requirements for a quality dashboard: lessons from national clinical audits. AMIA Annu Symp Proc 2020; 2019: 735-744
• 5 Burningham Z, Jackson GL, Kelleher J. et al. The enhancing quality of prescribing practices for older veterans discharged from the emergency department (EQUIPPED) potentially inappropriate medication dashboard: a suitable alternative to the in-person academic detailing and standardized feedback reports of traditional EQUIPPED?. Clin Ther 2020; 42 (04) 573-582
• 6 Dowding D, Merrill J, Russell D. Using feedback intervention theory to guide clinical dashboard design. AMIA Annu Symp Proc 2018; 2018: 395-403
• 7 Twohig PA, Rivington JR, Gunzler D, Daprano J, Margolius D. Clinician dashboard views and improvement in preventative health outcome measures: a retrospective analysis. BMC Health Serv Res 2019; 19 (01) 475
• 8 Koopman RJ, Kochendorfer KM, Moore JL. et al. A diabetes dashboard and physician efficiency and accuracy in accessing data needed for high-quality diabetes care. Ann Fam Med 2011; 9 (05) 398-405
• 9 Fischer MJ, Kourany WM, Sovern K. et al. Development, implementation and user experience of the Veterans Health Administration (VHA) dialysis dashboard. BMC Nephrol 2020; 21 (01) 136
• 10 Greco KJ, Rao N, Urman RD, Brovman EY. A dashboard for tracking mortality after cardiac surgery using a national administrative database. Cardiol Res 2021; 12 (02) 86-90
• 11 Makic MBF, Stevens KR, Gritz RM. et al. Dashboard design to identify and balance competing risk of multiple hospital-acquired conditions. Appl Clin Inform 2022; 13 (03) 621-631
• 12 Bersani K, Fuller TE, Garabedian P. et al. Use, perceived usability, and barriers to implementation of a patient safety dashboard integrated within a vendor EHR. Appl Clin Inform 2020; 11 (01) 34-45
• 13 Ivers NM, Sales A, Colquhoun H. et al. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014; 9: 14
• 14 Foster M, Albanese C, Chen Q. et al. Heart failure dashboard design and validation to improve care of veterans. Appl Clin Inform 2020; 11 (01) 153-159
• 15 Hartzler AL, Chaudhuri S, Fey BC, Flum DR, Lavallee D. Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design. EGEMS (Wash DC) 2015; 3 (02) 1133
• 16 Murphy DR, Savoy A, Satterly T, Sittig DF, Singh H. Dashboards for visual display of patient safety data: a systematic review. BMJ Health Care Inform 2021; 28 (01) e100437
• 17 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
• 18 Risling TL, Risling DE. Advancing nursing participation in user-centred design. J Res Nurs 2020; 25 (03) 226-238
• 19 Kramer BJ, Creekmur B, Howe JL. et al. Veterans affairs geriatric scholars program: enhancing existing primary care clinician skills in caring for older veterans. J Am Geriatr Soc 2016; 64 (11) 2343-2348
• 20 Knudsen SV, Laursen HVB, Johnsen SP, Bartels PD, Ehlers LH, Mainz J. Can quality improvement improve the quality of care? A systematic review of reported effects and methodological rigor in plan-do-study-act projects. BMC Health Serv Res 2019; 19 (01) 683
• 21 Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf 2014; 23 (04) 290-298
• 22 Kane DW, Hohman MM, Cerami EG, McCormick MW, Kuhlmman KF, Byrd JA. Agile methods in biomedical software development: a multi-site experience report. BMC Bioinformatics 2006; 7: 273
• 23 Tumosa N, Horvath KJ, Huh T. et al. Health care workforce development in rural America: when geriatrics expertise is 100 miles away. Gerontol Geriatr Educ 2012; 33 (02) 133-151
• 24 Esquer Rochin MA, Gutierrez-Garcia JO, Rosales JH, Rodriguez LF. Design and evaluation of a dashboard to support the comprehension of the progression of patients with dementia in day centers. Int J Med Inform 2021; 156: 104617
• 25 U.S. Department of Health and Human Services. The Research-Based Web Deisgn & Usability Guidelines, Enlarged/Expanded edition. Washington: US Government Printing Office; 2006
• 26 Richter Lagha R, Burningham Z, Sauer BC. et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020; 11 (04) 528-534
• 27 By the American Geriatrics Society Beers Criteria Update Expert P. American Geriatrics Society 2019 Updated AGS Beers Criteria(R) for Potentially Inappropriate Medication Use in Older Adults. J Am Geriatr Soc 2019; 67 (04) 674-694
• 28 van Deen WK, Cho ES, Pustolski K. et al. Involving end-users in the design of an audit and feedback intervention in the emergency department setting – a mixed methods study. BMC Health Serv Res 2019; 19 (01) 270
• 29 Tuti T, Nzinga J, Njoroge M. et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci 2017; 12 (01) 61
• 30 Brehaut JC, Colquhoun HL, Eva KW. et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med 2016; 164 (06) 435-441
• 31 Lu CY, Vinci LM, Quinn MT, Wilkes AE, Chin MH, Peek ME. Using feedback to change primary care physician behavior. J Ambul Care Manage 2015; 38 (02) 118-124
• 32 Yarahuan JKW, Lo HY, Bass L, Wright J, Hess LM. Design, usability, and acceptability of a needs-based, automated dashboard to provide individualized patient-care data to pediatric residents. Appl Clin Inform 2022; 13 (02) 380-390
• 33 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422

Publication History

Received: 20 May 2022

Accepted: 05 August 2022

Article published online:
12 October 2022