Heart Failure Dashboard Design and Validation to Improve Care of Veterans

Abstract

Background Early electronic identification of patients at the highest risk for heart failure (HF) readmission presents a challenge. Data needed to identify HF patients are in a variety of areas in the electronic medical record (EMR) and in different formats.

Objective The purpose of this paper is to describe the development and data validation of a HF dashboard that monitors the overall metrics of outcomes and treatments of the veteran patient population with HF and enhancing the use of guideline-directed pharmacologic therapies.

Methods We constructed a dashboard that included several data points: care assessment need score; ejection fraction (EF); medication concordance; laboratory tests; history of HF; and specified comorbidities based on International Classification of Disease (ICD), ninth and tenth codes. Data validation testing with user test scripts was utilized to ensure output accuracy of the dashboard. Nine providers and key senior management participated in data validation.

Results A total of 43 medical records were reviewed and 66 HF dashboard data discrepancies were identified during development. Discrepancies identified included: generation of multiple EF values on a few patients, missing or incorrect ICD codes, laboratory omission, incorrect medication issue dates, patients incorrectly noted as nonconcordant for medications, and incorrect dates of last cardiology appointments. Continuous integration and builds identified defects—an important process of the verification and validation of biomedical software. Data validation and technical limitations are some challenges that were encountered during dashboard development. Evaluations by testers and their focused feedback contributed to the lessons learned from the challenges.

Conclusion Continuous refinement with input from multiple levels of stakeholders is crucial to development of clinically useful dashboards. Extraction of all relevant information from EMRs, including the use of natural language processing, is crucial to development of dashboards that will help improve care of individual patients and populations.

 

MissingLinkText

Keywords

predictive analytics - decision support - dashboard - heart failure - population metrics

Background and Significance

Heart failure (HF), a prevalent and costly condition, is one of the leading causes for hospitalization in the United States among adults[1] [2] and is the most common Medicare inpatient discharge diagnosis.[3] Within the Veterans Affairs (VA) healthcare system, HF is the second most common diagnosis as well as one of the most costly diagnoses to treat annually.[4] [5] [6] [7] The relatively higher prevalence of HF[8] [9] among veterans may be secondary to their elevated risk of poor physical and mental health.[10] [11] [12] [13] The Veterans Integrated Service Network (VISN 1) has eight medical centers located in six New England states that deliver care to approximately 4,200 veterans with HF. This subset of the population serviced by VISN 1 has serious cardiopulmonary disease, as well as the highest readmission rates.[14] Studies investigating reasons for these high readmission rates identified medication discrepancies and cognitive impairment as likely contributors.[15] [16] One study noted that 30-day readmission rates were higher in HF patients who were not on a target dose of β blockers or vasodilators.[9]

However, early electronic identification of patients at the highest risk for admission presents a challenge.[17] The data needed to identify HF patients are located in a variety of areas in the electronic medical record (EMR) and in different formats (free text notes, images, and coded data). Dashboards for population health management are powerful tools that can be used to identify subsets of patients to improve their care and track their progress toward performance goals.[18] [19] The use of dashboards within the VA has enabled the ability to implement system-wide processes for both population management and quality improvement.[18] [20] [21]

Dashboards that utilize both structured and unstructured data can produce patient-specific risk assessments and support guideline-directed medical therapy (GDMT) recommendations leading to greater evidence-based care.[22] [23] [24] Previous studies have demonstrated the efficacy of using multiple identifiers in algorithms or from notes using natural language processing (NLP) for identifying patient phenotype for early diagnosis of HF.[25] [26] [27] [28] We therefore endeavored to develop and implement a dashboard that tracks clinical treatment. We hypothesized that the dashboard would help to identify all patients with HF that are at the highest risk for readmission across the medical center with the hopes of supporting better patient care and improving the cost-effectiveness of clinical practice, which is a major goal of the VA.[29]

 

MissingLinkText

Objective

The purpose of this paper is to describe the development and data validation of a HF dashboard that monitors the overall metrics of outcomes and treatments of the veteran patient population with HF while providing guidance to clinicians on mainstay of pharmacologic therapies.

 

MissingLinkText

Methods

The process improvement project took place at VA Boston Medical Center, a 150-bed tertiary care hospital that provides care to veterans from four of the six New England states within VISN 1. The initial project team of a business analyst, database programmer, and a cardiologist with expertise in HF met to discuss the dashboard's development and determined that an Agile software development approach[30] would be most feasible. The team collaborated with the Director of Primary Care, Chief Medical Officer, and Primary Care Patient Aligned Care Teams (PACTs) who would be the primary users of the dashboard. PACTs consist of providers, nurse care managers, clinical associates, pharmacists, and ancillary staff that manage Veteran patients' overall health care.[31] Key Performance Indicators (KPIs) and data points used to develop the dashboard were determined by the cardiologist and refined with input from the PACTs, Director of Primary Care, and Chief Medical Officer. The KPIs evaluate effectiveness of HF management across VISN 1: admissions per 100 patients by fiscal year; admissions for HF per 100 patients by fiscal year; bed day of care per 100 patients by fiscal year; and ER visit per 100 patients by fiscal year. These KPIs were based on the quality metrics for HF care that now exist based on process of care and outcomes.[1] [32]

Data points

Data points were obtained from three different databases and included: care assessment need (CAN) score,[33] [34] EF, medication history, specific laboratory testing (creatinine, B-type natriuretic peptide, hemoglobin, and sodium), specific comorbidities (chronic kidney disease, chronic obstructive pulmonary disease, diabetes, and coronary artery disease), primary care, and cardiology appointments. The CAN score is calculated by the Veterans Health Administration Support Service Center. It reflects the estimated probability of three outcomes for an individual veteran patient: (1) hospitalization, (2) death, and (3) hospitalization or death. The percentile of these probabilities ranges from 0 (lowest risk of hospitalization or death) to 99 (highest risk hospitalization or death), and it gives a perspective on how the patient compares with other veterans in terms of their likelihood of a given event over a 90-day period.[33] NLP and information extraction techniques were used to extract EF data from free text notes. Based on the literature, four distinct EF cohorts were formed: HF with preserved EF (HFpEF) >50%, HF with reduced EF (HFrEF) ≤40%, HF with recovered EF (HFrecEF), patients who had an EF at one time of <40%, and is now >50%, and HF with intermediate or mid-range EF (HFmrEF) 41 to 49%.[1] Querying of structured data was done by the database programmer who first confirmed a HF diagnosis using International Classification of Disease (ICD) 9 code of 428.x or an ICD 10 code of 150.x. An algorithm was used to determine patients' prescription concordance which is the agreement between the provider's treatment plan and current GDMT.[1] [35] The algorithm searched the records to see if any of the target medications ([Table 1]) were ordered in the past 5 years. In addition to prescription concordance, medication concordance was evaluated. Medication concordance provides information on whether patients are on target doses of GDMT. The algorithm determined patients' medication concordance for each class of medication listed in [Table 1] and noted if there was full concordance, partial concordance, partially concordant but not at target dose or nonconcordant. Fully concordant are those on target doses of medications noted in [Table 1]. Partially concordant are those on at least one class of medication at target dose while partially concordant but not at target dose(s) are those on all classes of medications, but none are at target dose. Nonconcordant refers to those not taking any recommended medications.

Table 1 Prescriptions and respective target dose

Medication class	Prescription and target dose
β blockers	• Metoprolol succinate target dose 200 mg daily • Carvedilol target dose 25 mg twice a day • Bisoprolol target dose 10 mg daily
Vasodilators	• Lisinopril, or fosinopril, or enalapril target dose 20 mg daily • Captopril target dose 50 mg three times a day • Valsartan target dose 160 mg twice a day • Candesartan target dose 32 mg daily • Losartan 150 mg daily • Hydralazine 300 mg daily and isosorbide 160 mg daily (both prescriptions)
Aldosterone blockers	• Spironolactone 25 mg daily • Epleronone 25 mg daily
Diuretics (no specific dosage or frequency)	• Furosemide • Bumetanide • Torsemide • Metolazone • Chlorthalidone

 

MissingLinkText

Data Validation

Members of the PACTs conducted data validation testing at three different points during the dashboard development. They were given a test guide and scripts which were used to validate information from excel spreadsheets containing graphs, dashboard table, and patient detail reporting for a specific site and clinician. Validation testers were directed to indicate “P” for pass if the test case presented the expected result in each field or “F” for fail if that expectation was not met.

 

MissingLinkText

 

MissingLinkText

Results

We developed the dashboard and completed validation testing in March of 2019. The landing page of the HF dashboard has four graphs see [Fig. 1]. All graphs display outcomes for each medical center within VISN 1. The landing page also has four tables that show the patient base for each of the four defined cohorts by EF level [Fig. 2]. Note that medication concordance is not displayed for patients in the HFpEF and HF with intermediate EF cohort as no current guidelines exists for patients with these EFs. The medication concordance algorithm will re-run on a nightly basis to capture changes in patient's medications as patients have the potential to transition to another cohort if they experience a major cardiovascular event. A drill down report of individual patients is available to clinicians by clicking on their respective home Veteran Affairs Medical Center from one of the four cohort-based tables noted in [Fig. 2]. [Fig. 3] is an example of a drill down report that is populated with fictitious data. The report displays in a left to right with color coded columns to enhance the ease of reading and includes a hyperlink to GDMT for providers to review if necessary.

Fig. 1 Graphs on dashboard home screen.

Fig. 2 Tables on dashboard home screen.

Fig. 3 Example of a drill down reduced EF cohort report (fictional patient data). EF, ejection fraction.

Eleven clinicians in total, seven physicians, two nurse practitioners, and two nurses validated dashboard data by comparing it with patient EMRs. A total of 43 medical records were reviewed and 66 HF dashboard data discrepancies or issues were identified [Table 2]. In addition to the discrepancies, users also provided suggestions on how to increase usability of the dashboard. Suggestions included the flow of information within the dashboard, the use of color to differentiate columns in the drill-down reports page as well as changes in wording, for example, compliance was replaced by concordance and we modified cohort definitions to make them clearer.

Table 2 Results of testing over time

	Time I charts reviewed (n = 18) Number of providers (n = 5)	Time II charts reviewed (n = 11) Number of providers (n = 4)	Time III charts reviewed (n = 14) Number of providers (n = 2)
Issue/discrepancy	Number of frequencies of issue/discrepancy over time
Missing Hgb value	6	3
Inaccuracy of last EF note date	7	3	10
CKD, CAD, and DM diagnosis not captured	14	5
BNP value missing		1
Patients concordant for medications in chart, but it did not show up on dashboard	7	1
Cardiology was not noted as the specialty care	1
Furosemide issue date was incorrect	3
Metoprolol issue date was incorrect	1
Spironolactone issue date was incorrect	1
Cardio Appt within last 1 year was incorrect			2
Deceased patient included in the dashboard population			1

Abbreviations: BNP, brain natriuretic peptide; CAD, coronary artery disease; CKD, chronic kidney disease; DM, diabetes mellitus; EF, ejection fraction; Hgb, hemoglobin.

 

MissingLinkText

Discussion

For this process improvement project, an Agile software development approach was used to carefully explore and integrate the perspectives of key stakeholders into the dashboard. Dashboards developed with input from end-users, leadership, and subject matter experts have a greater chance of being adopted and have higher user acceptance.[21] [36] [37] The project team used structured and unstructured VA data to develop the dashboard. The dashboard presents data via a clear mechanistic interface and allows users to see comparisons between hospitals. The HF dashboard is suitable for the use case because it will allow for collaborative population care among members of primary care PACTs. It can also be used by pharmacy to review prescriptions concordance and collaborate with primary care/cardiology for appropriate follow-up.

During dashboard development, we encountered several challenges, duplicate data, missing data, inaccuracy in the last EF note date (the date when the note was added to EMR, not necessarily the actual day the echocardiogram was performed), and multiple EFs noted on the same patient in a single date. This occurred because the dashboard reports the date of the note containing an EF value and not the date when the echocardiogram was performed. It should be noted that although EF data are initially important to first categorize patients into a cohort, once patients are placed in a cohort the exact dates of the EF data update and last EF note date are unlikely to affect the HF cohort designation. Multiple EFs in a single note were encountered by the NLP algorithm which led to several EFs being generated for a few patients. The issue was discovered and adjustments were made to the algorithm used to cleanse the data to ensure that the algorithm would handle similar future cases appropriately by choosing a truly representative EF value. Furthermore, re-running of the algorithm will occur approximately every 6 months to capture EF changes over that time period. This will ensure that the newest EF is represented in the dashboard. Another issue involved deceased patients populating in the dashboard. Data are uploaded nightly to the VA's corporate data warehouse and therefore if a patient's status is listed as alive that person will populate in the dashboard but will automatically be removed from the dashboard, once the system refreshes at the next upload. With the validation of the data elements, we see great potential for the dashboard to enhance HF care at VA Boston. Overall, evidence indicates that implementing healthcare dashboards can improve clinician adherence to quality guidelines and more consistently provide GDMT[38] and may play an important role in decreasing readmission rates.[39] Furthermore, if used consistently the HF dashboard could encourage discussions between providers and patients regarding appropriate HF treatment goals. However, it is recognized that introducing dashboards can impact workflow.[40] Therefore, the goal is to continue to seek feedback from users on how to improve the dashboard's usefulness, information quality and efficiency[40] to ensure that the dashboard leverages data that informs clinicians on how best to manage HF patients.

 

MissingLinkText

Limitations

One major drawback is that the current structure of the EMR does not support either embedding or placing a direct link to the dashboard into the EMR. However, primary care clinicians agreed to add the dashboard to their hub which can be accessed via a link from the EMR. The data hub is a location where common data tools, reports, and dashboards are stored for use by primary care staff to perform their duties. Another limitation is that the dashboard does not capture prescriptions written and filled by providers outside of the VA. To capture this information, both cardiology and primary care providers collect this information from patients and place it in the EMR.

 

MissingLinkText

Conclusion

This paper presents a HF dashboard providing real-time information to support better patient care and thereby improve population metrics. Healthcare dashboards that utilize both structured and unstructured data, such as the HF Dashboard, can provide cohort-based groupings as well as individual patient-based risk assessments to help primary care providers identify and appropriately treat HF patients according to GDMT. Overall, there are several challenges and opportunities that come with using the HF dashboard. The collaboration between primary care and cardiology to accomplish the shared goal of increasing access to quality HF healthcare is central to the success of this project. Future work with this dashboard will involve follow-up user testing to evaluate the tool's usefulness, usability, and effectiveness by a group of primary care providers and their PACTs. In addition, process of care and outcomes will be reassessed at the 1-year mark to determine the quality of HF care within VISN 1.

 

MissingLinkText

Clinical Relevance Statement

This case report has clinical relevance for clinicians, programmers, and quality improvement staff because it adds to the evidence that developing dashboards with input from end-users enhances the willingness to adopt new software.

 

MissingLinkText

Multiple Choice Questions

1. Which of the following are best practices of data validation testing?

   • Compare the output result with the expected.

   • Test on full complete data instead of sample data.

   • No need to have a detailed plan as things will change along the way.

   • Handle bad data incorrectly.

   Correct Answer: The correct answer is a, compare the output result with the expected. One challenge in our improvement process project was dealing with the inaccurate outputs over the development process and determining the root cause. Because validation testers alerted us to the incorrect output, we were able to remedy the issues.

2. When developing dashboards to support a process improvement project, besides an SQL database programmer who should be a partner in the design?

   • A postdoctoral medical informatics fellow who is completing the last month of their fellowship.

   • The chief informatics officer.

   • The end user.

   • Another database programmer.

   Correct Answer: The correct answer is option c, the end user. This is crucial because they are the people who will use the software daily. Our project used an Agile approach to ensure that the stakeholders who were the end users not only gave input to the dashboard during development but also validated the data used in the dashboard.

 

MissingLinkText

 

MissingLinkText

Conflict of Interest

None declared.

Authors' Contributions

All authors meet the requirements for authorship and manuscript submission.

Protection of Human and Animal Subjects

This project was approved and given nonresearch designation by the VISN 1 Research and Development committee.

• References

• 1 Yancy CW, Jessup M, Bozkurt B. , et al. 2013 ACCF/AHA guideline for the management of heart failure: executive summary: a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. Circulation 2013; 128 (16) 1810-1852
• 2 Mozaffarian D, Benjamin EJ, Go AS. , et al; American Heart Association Statistics Committee and Stroke Statistics Subcommittee. Heart disease and stroke statistics--2015 update: a report from the American Heart Association. Circulation 2015; 131 (04) e29-e322
• 3 CMS. National summary of inpatient charge data by Medicare severity diagnosis related group (MS-DRG), FY2016. 7500 Security Boulevard, Baltimore, MD 21244: Centers for Medicare & Medicaid Services. Available at: https://data.cms.gov/Medicare-Inpatient/National-Summary-of-Inpatient-Charge-Data-by-Medic/sfua-yggc . Accessed August 14, 2018
• 4 Nuti SV, Qin L, Rumsfeld JS. , et al. Association of admission to veterans affairs hospitals vs non-veterans affairs hospitals with mortality and readmission rates among older men hospitalized with acute myocardial infarction, heart failure, or pneumonia. JAMA 2016; 315 (06) 582-592
• 5 Zulman DM, Pal Chee C, Wagner TH. , et al. Multimorbidity and healthcare utilisation among high-cost patients in the US veterans affairs health care system. BMJ Open 2015; 5 (04) e007771
• 6 Chapko MK, Liu CF, Perkins M, Li YF, Fortney JC, Maciejewski ML. Equivalence of two healthcare costing methods: bottom-up and top-down. Health Econ 2009; 18 (10) 1188-1201
• 7 Krishnamurthi N, Francis J, Fihn SD, Meyer CS, Whooley MA. Leading causes of cardiovascular hospitalization in 8.45 million US veterans. PLoS ONE 2018; 13 (03) e0193996
• 8 Jones LG, Sin M-K, Hage FG. , et al. Characteristics and outcomes of patients with advanced chronic systolic heart failure receiving care at the Veterans Affairs versus other hospitals: insights from the Beta-blocker Evaluation of Survival Trial (BEST). Circ Heart Fail 2015; 8 (01) 17-24
• 9 Silanskas G, Bergman CL, Kaplan BJ. Factors affecting heart failure readmission rates in VA patients. Fed Pract 2014; 31 (11) 14-20
• 10 Assari S. Veterans and risk of heart disease in the United States: a cohort with 20 years of follow up. Int J Prev Med 2014; 5 (06) 703-709
• 11 Brown AD, Antonius D, Kramer M, Root JC, Hirst W. Trauma centrality and PTSD in veterans returning from Iraq and Afghanistan. J Trauma Stress 2010; 23 (04) 496-499
• 12 Findley P, Shen C, Sambamoorthi U. Multimorbidity and persistent depression among veterans with diabetes, heart disease, and hypertension. Health Soc Work 2011; 36 (02) 109-119
• 13 Yoon J, Zulman D, Scott JY, Maciejewski ML. Costs associated with multimorbidity among VA patients. Med Care 2014; 52 (03) (Suppl. 03) S31-S36
• 14 Reese RL, Clement SA, Syeda S. , et al. Coordinated-transitional care for veterans with heart failure and chronic lung disease. J Am Geriatr Soc 2019; 67 (07) 1502-1507
• 15 Paquin AM, Salow M, Rudolph JL. Pharmacist calls to older adults with cognitive difficulties after discharge in a Tertiary Veterans Administration Medical Center: a quality improvement program. J Am Geriatr Soc 2015; 63 (03) 571-577
• 16 Rebello KE, Gosian J, Salow M, Sweeney P, Rudolph JL, Driver JA. The Rural PILL Program: a postdischarge telepharmacy intervention for rural veterans. J Rural Health 2017; 33 (03) 332-339
• 17 Cox ZL, Lewis CM, Lai P, Lenihan DJ. Validation of an automated electronic algorithm and “dashboard” to identify and characterize decompensated heart failure admissions across a medical center. Am Heart J 2017; 183: 40-48
• 18 Carmichael JM, Meier J, Robinson A, Taylor J, Higgins DT, Patel S. Leveraging electronic medical record data for population health management in the Veterans Health Administration: successes and lessons learned. Am J Health Syst Pharm 2017; 74 (18) 1447-1459
• 19 Nelson O, Sturgis B, Gilbert K. , et al. A visual analytics dashboard to summarize serial anesthesia records in pediatric radiation treatment. Appl Clin Inform 2019; 10 (04) 563-569
• 20 Adams SL. IT dashboard reduces high-dose opioid prescriptions among Veterans. O & P News; . Available at: https://www.healio.com/psychiatry/addiction/news/online/%7B7bcb7dcd-e538-4da8-8685-2429bf93c852%7D/it-dashboard-reduces-high-dose-opioid-prescriptions-among-veterans . Accessed July 2015
• 21 Lau MK, Bounthavong M, Kay CL, Harvey MA, Christopher MLD. Clinical dashboard development and use for academic detailing in the U.S. Department of Veterans Affairs. J Am Pharm Assoc (2003) 2019; 59 (2s): S96-S103
• 22 Garg AX, Adhikari NK, McDonald H. , et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 23 Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007; 14 (02) 141-145
• 24 Smith MW, Brown C, Virani SS. , et al. Incorporating guideline adherence and practice implementation issues into the design of decision support for beta-blocker titration for heart failure. Appl Clin Inform 2018; 9 (02) 478-489
• 25 Byrd RJ, Steinhubl SR, Sun J, Ebadollahi S, Stewart WF. Automatic identification of heart failure diagnostic criteria, using text analysis of clinical notes from electronic health records. Int J Med Inform 2014; 83 (12) 983-992
• 26 Pakhomov S, Weston SA, Jacobsen SJ, Chute CG, Meverden R, Roger VL. Electronic medical records for clinical research: application to the identification of heart failure. Am J Manag Care 2007; 13 (6 Part 1): 281-288
• 27 Shivade C, Raghavan P, Fosler-Lussier E. , et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. J Am Med Inform Assoc 2014; 21 (02) 221-230
• 28 Son CS, Kim YN, Kim HS, Park HS, Kim MS. Decision-making model for early diagnosis of congestive heart failure using rough set and decision tree approaches. J Biomed Inform 2012; 45 (05) 999-1008
• 29 VHA OoRaD. VA Quality Enhancement Research Initiative (QUERI) Strategic Plan 2016–2020. Available at: https://www.queri.research.va.gov/about/strategic_plans/QUERIStrategicPlan.pdf . Accessed January 9, 2020
• 30 Schmidt J, Weyrauch K. Getting ‘agile’ with medical device development. Biomed Instrum Technol 2013; 47 (03) 221-223
• 31 Department of Veterans A. Patient Aligned Care Team (PACT) Handbook. Washington, DC 20420: Department of Veterans Affairs, Veterans Health Administration; 1101.10(1). Available at: http://docplayer.net/15987660-Department-of-veterans-affairs-vha-handbook-1101-10-washington-dc-20420-february-5-2014-patient-aligned-care-team-pact-handbook.html . Accessed February 5, 2014
• 32 Bonow RO, Ganiats TG, Beam CT. , et al; American Academy of Family Physicians; American Academy of Hospice and Palliative Medicine; American Nurses Association; American Society of Health-System Pharmacists; Heart Rhythm Society; Society of Hospital Medicine; ACCF/AHA Task Force on Performance Measures. ACCF/AHA/AMA-PCPI 2011 performance measures for adults with heart failure: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures and the American Medical Association-Physician Consortium for Performance Improvement. J Am Coll Cardiol 2012; 59 (20) 1812-1832
• 33 Update on the Care Assessment Need Score – CAN 2.0 and the Patient Care Assessment System (PCAS) [Webinar]. Virtual2016. Available at: https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=1088 . Accessed January 9, 2020
• 34 Wang L, Porter B, Maynard C. , et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care 2013; 51 (04) 368-373
• 35 Yancy CW, Jessup M, Bozkurt B. , et al. 2017 ACC/AHA/HFSA focused update of the 2013 ACCF/AHA Guideline for the management of heart failure: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart Failure Society of America. J Am Coll Cardiol 2017; 70 (06) 776-803
• 36 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
• 37 Ahmed R, Toscos T, Rohani Ghahari R. , et al. Visualization of cardiac implantable electronic device data for older adults using participatory design. Appl Clin Inform 2019; 10 (04) 707-718
• 38 Dowding D, Randell R, Gardner P. , et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015; 84 (02) 87-100
• 39 Patel J, Sandhu A, Parizo J, Moayedi Y, Fonarow GC, Heidenreich PA. Validity of performance and outcome measures for heart failure. Circ Heart Fail 2018; 11 (09) e005035
• 40 Schleyer TKL, Rahurkar S, Baublet AM. , et al; FHIR Development Team. Preliminary evaluation of the Chest Pain Dashboard, a FHIR-based approach for integrating health information exchange information directly into the clinical workflow. AMIA Jt Summits Transl Sci Proc 2019; 2019: 656-664