Timely Data for Targeted Quality Improvement Interventions: Use of a Visual Analytics Dashboard for Bronchiolitis

Abstract

Background Standard methods for obtaining data may delay quality improvement (QI) interventions including for bronchiolitis, a common cause of childhood hospitalization.

Objective To describe the use of a dashboard in the context of a multifaceted QI intervention aimed at reducing the use of chest radiographs, bronchodilators, antibiotics, steroids, and viral testing in patients with bronchiolitis.

Methods This QI initiative took place at Children's Minnesota, a large, not-for-profit children's health care organization. A multidisciplinary bronchiolitis workgroup developed a local clinical guideline and order-set. Delays in obtaining baseline data prompted a pediatric hospitalist and information technology specialist to modify a vendor's dashboard to display data related to bronchiolitis guideline metrics. Patients 2 months to 2 years old with a bronchiolitis emergency department (ED)/inpatient encounter in the period October 1, 2014 to April 30, 2018 were included. The primary outcome was a functioning dashboard; a process measure was the percentage of ED clinician logins. Outcome measures included the percent use of guideline metrics (e.g., bronchodilators) displayed on statistical process control charts (ED vs. inpatient). Balancing measures included length of stay, charge ratios, and hospital revisits.

Results A workgroup (formed October 2015) implemented a bronchiolitis order-set and guideline (February 2016) followed by a bronchiolitis dashboard (August 2016) consolidating disparate data sources loaded within 2 to 4 days of discharge. In total, 35% of ED clinicians logged in. Leaders used the dashboard to target and track interventions such as a bronchodilator order alert. There were improvements in most outcome metrics; however, timing did not suggest direct dashboard impact. ED balancing measures were lower after implementation.

Conclusion We described use of a dashboard to support a multifaceted QI initiative for bronchiolitis. Leaders used the dashboard for targeted interventions but the dashboard did not directly impact the observed improvements. Future studies should assess reasons for low individual dashboard use.

Keywords

dashboard - guideline - performance

Background and Significance

Quality improvement (QI) targets unwarranted variations in care.[1] National guidelines for bronchiolitis, the fourth most common reason for pediatric hospitalization in the United States, encourage limiting disproven treatments (e.g., antibiotics).[2] [3] QI initiatives for bronchiolitis have proven effective[4] [5] [6] [7] [8] [9] [10] [11] [12] but rely on timely data for audit and feedback.[13] [14] Traditional methods to obtain data may be challenging in some organizations.[15] [16]

At our organization we had several challenges contributing to variation in care for bronchiolitis, including (1) lack of accessible clinical guidelines, (2) minimal data support for QI initiatives, and (3) lack of organizational process for providing feedback to clinicians. In 2015 we joined a national QI initiative, “Stewardship in Improving Bronchiolitis,” from the American Academy of Pediatrics (AAP)[12] which provided tools and coaching to a local bronchiolitis QI workgroup and required close tracking of data to design targeted interventions. Our initial data request through the organization's data warehouse took 6 months to be completed, prompting us to review alternate options for acquiring data.

Visual analytics dashboards are one mechanism to overcome repeated manual electronic health record (EHR) queries to support QI integrating data into a user interface enabling tracking, planning, and comparisons with near real-time data from the EHR and other sources.[17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] Users may modify inclusions/exclusions to focus on their population of interest without repeat data requests.[28] [29] [30]

Objective

We aimed to describe the use of a visual analytics dashboard to support a multifaceted QI initiative for patients with bronchiolitis. We also intended to determine if the dashboard directly impacted the success toward aims of the multifaceted QI initiative, which were to achieve 20% reductions in the use of chest X-rays, bronchodilators, antibiotics, steroids, and viral testing in patients with bronchiolitis by April 2018.

Methods

The QI initiative took place at Children's Minnesota (CM), a large, independent, not-for-profit, tertiary children's health care organization with approximately 14,000 inpatient and 96,000 emergency department (ED) visits annually. There were five key interventions to the QI initiative. First, we joined a national bronchiolitis QI collaborative,[12] and second gathered a CM bronchiolitis workgroup (October 2015)—including representatives from ED, hospitalists, nursing, respiratory therapy, information technology (IT), pharmacy, and critical care—to determine local QI interventions. Third, clinicians received education (e.g., evidence behind limiting bronchodilators) at staff meetings and CM Grand Rounds January and October 2016. Nurses and respiratory therapists received education via modules and e-mail newsletters. Fourth, (February 2016) we published a local modification of the AAP 2014 bronchiolitis guideline[2] and a companion order-set to our intranet and EHR (Cerner[32]).

Following local QI interventions there were anecdotal improvements but data delays emerged as a key barrier to success; data requested in March 2016 was delivered in August 2016. As data turnaround was crucial for targeted interventions we sought an alternate method to obtain data. Workgroup members had used operational dashboards[33] (implemented 2011) and felt a clinical dashboard might improve bronchiolitis data procurement.

As a fifth QI intervention a pediatric hospitalist (bronchiolitis workgroup lead) and an IT dashboard developer partnered in modifying a vendor's analytic dashboard[33] for use in bronchiolitis. They determined target patient population based upon previous guidelines/studies,[2] [34] categorized clinicians (ED vs. observation/inpatient [hereafter referred to as inpatient]), categorized tests/treatments (e.g., medications considered “antibiotics”), determined display metrics, benchmarks,[34] and verified data accuracy over ∼6 months. The hospitalist monitors the dashboard monthly during bronchiolitis season and works with the IT dashboard developer to resolve data accuracy or display issues.

Organizational leaders were granted dashboard access; feedback from early users was collected informally over 1 month and resulted in minor changes. Issues with accuracy of individual hospitalist data related to resident order entry led to limitation of individual log-ins to ED clinicians and organizational leaders.

The primary objective of this case study was to describe use of a bronchiolitis dashboard in the context of a multifaceted QI initiative. The process measure was the percentage of individual ED clinicians who logged in to the dashboard, obtained from IT records. QI outcome measures were percent use of chest radiographs, bronchodilators, antibiotics, steroids, and viral testing.[2] Balancing measures included length of stay (LOS), charge (hospital facility charges and professional fees presented as ratios per CM policy), and 7-day same-cause ED revisits or hospital readmissions.

Patients 2 months to 2 years old seen in ED/inpatient settings at CM with bronchiolitis (International Classification of Disease 9 or 10 codes 466.11, 466.19 or J21.x) in the period October 1, 2014 to April 30, 2018 were included. Data from May to September of each year were excluded as we suspected there may be appropriately higher use of nonrecommended tests/treatments outside of the typical bronchiolitis season and elected to focus QI interventions on peak season. Patients in the intensive care units or with a secondary diagnosis of asthma, pneumonia, or underlying complex chronic condition (including gestational age < 27 weeks)[35] were excluded.

We used statistical process control (SPC) p-charts for outcome measures using QI Macros (KnowWare International Inc., version 2018). October 2014 to April 2015 represented the baseline period for calculation of upper and lower control limits; a process change was indicated for the implementation period of October 2015 to April 2018 to calculate new control limits. We determined a priori[3] 20% reductions in the baseline metric mean to be clinically relevant, based upon our preimplementation QI aim. Measures were separated into ED/inpatient based upon ordering clinician. Resident/fellow orders are assigned to the attending clinician on record at the time of the order; this was not modifiable on the dashboard. Balancing measures were analyzed with chi-squared tests for dichotomous outcomes and independent t-tests for continuous outcomes using STATA version 13.0.[36] p-Values < 0.05 were statistically significant.

Clinicians were not required to review personal data compared with peers and there were no implications (e.g., financial) to their performance. There were no conflicts of interest. This study was deemed QI and exempt from further review by the CM's Institutional Review Board.

Results

We implemented a bronchiolitis dashboard in August 2016, see [Fig. 1]. Default settings restrict the dataset to the goal patient population; users can add a chart or change inclusions/exclusions to analyze specific populations/units. Green and red colors indicate performance better than or worse than the benchmark,[34] respectively. Target metrics can also be displayed by demographic categories, such as primary payer or race. Thirty-five percent (20/57) of ED clinicians logged in to the dashboard at least once.

Fig. 1 Sample views of the bronchiolitis visual analytic dashboard.

See [Fig. 2] for sample SPC charts for bronchodilator use with annotation of QI interventions; see online supplemental materials for SPC charts and summary table of percent change for all metrics ([Supplementary Figs. S1] and [S2], [Table S1], available in the online version). There was[3] 20% difference in all target metrics with the exception of ED and inpatient antibiotics and inpatient viral testing. For example, there was a shift in mean bronchodilator use from 66.7 to 43.8% in the ED and 72.1 to 46.4% in inpatients (33 and 36% reductions, respectively).

Fig. 2 (A, B) Statistical process control p-charts for bronchodilator use in patients with bronchiolitis. Baseline period: October 1, 2014 to April 30, 2015; implementation period: October 1, 2015 to April 30, 2018. CL, control limit (mean); LCL, lower control limit; UCL, upper control limit.

See [Table 1] for balancing measures. Comparing baseline with implementation periods, there were improvements in all ED balancing measures with a higher ED discharge rate (70.7 vs. 72.8%, p = 0.05), lower charges (ratio 1:0.86, p < 0.001), shorter LOS (2.9 vs. 2.6 hours, p = 0.001), and lower 7-day revisit rates (15.4 vs. 11.6%, p < 0.001). Inpatient charges increased (ratio 1:1.14, p = 0.01) but LOS and readmissions remained stable.

Table 1 Balancing measures across emergency department (ED) and observation/inpatient (admitted) care settings before (baseline) and after implementation of a multifaceted quality improvement initiative for patients with bronchiolitis

	Baseline (October 2014–April 2015)	Implementation (October 2015–April 2018)	p-Value
Discharged home from ED, n (%)	N = 2,035 (70.7)	N = 3,965 (72.8)	0.05
Mean ED length of stay, hours (95% CI)	2.9 (2.7–3.1)	2.6 (2.5–2.6)	<0.001
Mean ED charges, ratio	1	0.86	<0.001
7-Day ED revisit, n (%)	313 (15.4)	458 (11.6)	<0.001
Admitted, n (%)	N = 843 (29.3)	N = 1,485 (27.2)	0.05
Mean inpatient length of stay, days (95% CI)	2.3 (2.2–2.4)	2.6 (2.4–2.8)	0.07
Mean inpatient charges, ratio	1	1.14	0.01
7-Day readmission, n (%)	8 (0.95)	10 (0.67)	0.37

Abbreviation: CI, confidence interval.

Two targeted interventions resulted from the use of the dashboard. First, in response to high inpatient bronchodilator use noted in April 2016 (68%) we developed an EHR bronchodilator order alert (implemented mid-month, October 2016). Second, we exported individual scorecards from the dashboard to all ED clinicians in November 2017 with personal 2016 to 2017 data compared with peers. This process was repeated in late March 2018 with data from the 2017 to 2018 season. We were unable to determine which ED clinicians reviewed their scorecard.

Discussion

Health care dashboards have been used to improve workflow, reduce preventable harms, and track metrics across multiple sites.[17] [18] [20] [21] [22] [23] [24] [26] [27] [31] [37] We used a bronchiolitis clinical dashboard to support a multifaceted QI initiative for bronchiolitis. While improvement shifts were seen in most target metrics it is unlikely, based upon timing and low individual use, that these improvements were a direct result of the dashboard. The dashboard was viewed by leaders as instrumental in tracking adherence to guideline metrics and was used to inform additional targeted interventions.

The low individual log-in rate (35%) found in our study may reflect low team engagement, a key for QI success,[38] and may have tempered improvements in outcome measures. We suspect that because the dashboard required a separate login from the EHR, individual clinicians may have perceived low ease of use[39] [40] or were uninterested in their data.

Our data, as well as other QI studies without use of a dashboard, suggest that the other bronchiolitis QI interventions, such as the guideline and order-set, caused the observed improvements.[4] [5] [6] [7] [8] [9] [10] [11] [12] However, QI is reliant on having on timely data to perform Plan-Do-Study-Act cycles.[41] We experienced difficulty obtaining data (6 months) driving our decision to develop a dashboard where data are loaded to the dashboard within approximately 2 to 4 days of patient discharge, enabling near real-time feedback and targeted interventions. This dashboard allowed us to understand the impact of the other QI interventions.

While we saw significant shifts in all metrics aside from antibiotic use and inpatient viral testing, there was a relatively greater improvement in both outcome and balancing measures in the ED versus inpatients. It is possible that interventions targeted to ED clinicians alone (individual dashboard log-in, individual scorecards) would have contributed to this difference; however, we suspect this did not occur as the timing of ED improvements relative to the inpatient setting suggests that other factors may have been more important, such as shifts in patient severity evidenced by a higher rate of ED discharge in implementation. There may also have been differences in guideline uptake, lack of individual hospitalist feedback, or difficulty in changing culture.[6] [42] [43] [44] [45] Future studies should examine reasons for bronchiolitis guideline nonadherence.

There were no clinically relevant changes in antibiotic use in either setting (14% decrease in ED, 14% increase in inpatients). By using the dashboard to add the option to exclude patients with otitis media the antibiotic rate was under 5%, indicating alternate appropriate indications for high antibiotic use not captured by original data definitions.

Balancing measure improvements included shorter ED LOS and lower ED charges in the implementation periods. This is consistent with a previous study which found that higher adherence to a bronchiolitis guideline was associated with lower LOS and costs.[46] Inpatient LOS was unchanged and inpatient charges were slightly higher in our study. Our QI initiative did not target hospital LOS—which largely correlates with charges/costs in the inpatient setting—as such these findings were not surprising.

Future directions for this QI initiative include increasing clinician engagement and feedback and use of the dashboard to design and track new group goals (e.g., reducing inpatient viral testing). Future initiatives may apply heuristic evaluation to improve dashboard visualization[47] and explore barriers to use.

Challenges included attributing orders to appropriate clinicians in a teaching environment and a moderate learning curve. We found it critical to partner a clinician with the IT dashboard developer to determine default settings and understand workflow limitations. Due to low individual use it is difficult to determine the impact of individual performance review or the direct impact of the dashboard on outcomes. A final limitation to this study is that the cost of the QI initiative, including the dashboard which is subject to vendor negotiation, was not directly measured.

Conclusion

We described use of a visual analytics dashboard in a multifaceted bronchiolitis QI initiative. Subsequent to multiple QI interventions we reduced use of most nonrecommended tests and treatments in patients with bronchiolitis and improved ED balancing measures. However, timing of improvements and low individual clinician use suggest that the dashboard did not directly impact outcomes. The dashboard was helpful in overcoming organizational barriers to QI data procurement and in tracking the impact of other QI interventions and may be considered as a tool for other organizations with similar challenges.

Clinical Relevance Statement

This case report has relevance for clinicians, IT specialists, and QI specialists as it describes the use of a visual analytics dashboard to inform QI initiatives and improve clinical care.

Multiple Choice Questions

1. Which of the following may limit the ability to accurately attribute individual orders to individual attending clinicians on a visual analytics dashboard?

   • Clinician license type (e.g., MD vs. NP).

   • Attribution of resident orders.

   • Time of clinician order.

   • Type of order (e.g., medication vs. laboratory test).

   Correct Answer: The correct answer is option b, attribution of resident orders. One challenge in our QI initiative was how to deal with resident orders. The organization EMR assigns resident orders to the attending clinician of record at the time the order was placed. We found this to be accurate in the ED setting but not in the inpatient (e.g., hospitalist) setting where the attending provider may change multiple times in a day.

2. Who might be the best partner for an IT dashboard developer when creating a visual analytics dashboard to support a clinical QI initiative?

   • A resident on a 1-month QI elective.

   • Another software programmer within the IT department.

   • A front-line clinical leader such as an ED clinician or hospitalist with QI expertise.

   • The chief financial officer.

   Correct Answer: The correct answer is c, a front-line clinical leader such as an ED clinician or hospitalist with QI expertise. When bringing IT to the forefront of clinical work it is important to partner IT experts with front-line users and content experts. A lesson learned in our QI initiative was that it was crucial to have the IT developer partner with the QI clinical leader (a hospitalist) to review the dashboard, modify visual displays, and review data for validation.

Conflict of Interest

None declared.

Acknowledgments

The authors thank Dr. Emily Chapman and Madsen Zimbric for their reviews of the manuscript.

Authors' Contributions

Dr. Hester conceptualized and designed the study, led the implementation of the guideline, assisted in dashboard development, drafted the initial manuscript, performed statistical analyses, prepared several figures, and approved the final manuscript as submitted. Mr. Lang was the IT dashboard developer and led dashboard development, performed analyses, prepared several figures, critically reviewed the manuscript, and approved the final manuscript as submitted. Ms. Madsen and Dr. Tambyraja were IT leaders who assisted with organizational dashboard implementation, assisted in bronchiolitis dashboard development, participated in the guideline review, critically reviewed the manuscript, and approved the final manuscript as submitted. Dr. Zenker was an emergency department lead for the guideline implementation, assisted in dashboard development, critically reviewed the manuscript, and approved the final manuscript as submitted.

Protection of Human and Animal Subjects

There were no implications (e.g., financial) to clinicians as a result of this initiative. This study was deemed QI and exempt from further review by the organization's Institutional Review Board.

• References

• 1 Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999; 318 (7182): 527-530
• 2 Ralston SL, Lieberthal AS, Meissner HC. , et al; American Academy of Pediatrics. Clinical practice guideline: the diagnosis, management, and prevention of bronchiolitis. Pediatrics 2014; 134 (05) e1474-e1502
• 3 Keren R, Luan X, Localio R. , et al; Pediatric Research in Inpatient Settings (PRIS) Network. Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch Pediatr Adolesc Med 2012; 166 (12) 1155-1164
• 4 Akenroye AT, Baskin MN, Samnaliev M, Stack AM. Impact of a bronchiolitis guideline on ED resource use and cost: a segmented time-series analysis. Pediatrics 2014; 133 (01) e227-e234
• 5 Riese J, Porter T, Fierce J, Riese A, Richardson T, Alverson BK. Clinical outcomes of bronchiolitis after implementation of a general ward high flow nasal cannula guideline. Hosp Pediatr 2017; 7 (04) 197-203
• 6 Parikh K, Hall M, Teach SJ. Bronchiolitis management before and after the AAP guidelines. Pediatrics 2014; 133 (01) e1-e7
• 7 Mittal V, Hall M, Morse R. , et al. Impact of inpatient bronchiolitis clinical practice guideline implementation on testing and treatment. J Pediatr 2014; 165 (03) 570.e3-576.e3
• 8 Muething S, Schoettker PJ, Gerhardt WE, Atherton HD, Britto MT, Kotagal UR. Decreasing overuse of therapies in the treatment of bronchiolitis by incorporating evidence at the point of care. J Pediatr 2004; 144 (06) 703-710
• 9 Florin TA, Byczkowski T, Ruddy RM, Zorc JJ, Test M, Shah SS. Variation in the management of infants hospitalized for bronchiolitis persists after the 2006 American Academy of Pediatrics bronchiolitis guidelines. J Pediatr 2014; 165 (04) 786-792
• 10 Ralston S, Garber M, Narang S. , et al. Decreasing unnecessary utilization in acute bronchiolitis care: results from the value in inpatient pediatrics network. J Hosp Med 2013; 8 (01) 25-30
• 11 Ralston S, Comick A, Nichols E, Parker D, Lanter P. Effectiveness of quality improvement in hospitalization for bronchiolitis: a systematic review. Pediatrics 2014; 134 (03) 571-581
• 12 Mussman GM, Lossius M, Wasif F. , et al. Multisite emergency department inpatient collaborative to reduce unnecessary bronchiolitis care. Pediatrics 2018; 141 (02) e20170830
• 13 Gould NJ, Lorencatto F, Stanworth SJ. , et al. Application of theory to enhance audit and feedback interventions to increase the uptake of evidence-based transfusion practice: an intervention development protocol. Implement Sci 2014; 9: 92
• 14 Ivers N, Jamtvedt G, Flottorp S. , et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; (06) CD000259
• 15 Prokosch HU, Ganslandt T. Perspectives for medical informatics. Reusing the electronic medical record for clinical research. Methods Inf Med 2009; 48 (01) 38-44
• 16 Embi PJ, Payne PRO. Clinical research informatics: challenges, opportunities and definition for an emerging domain. J Am Med Inform Assoc 2009; 16 (03) 316-327
• 17 Shailam R, Botwin A, Stout M, Gee MS. Real-time electronic dashboard technology and its use to improve pediatric radiology workflow. Curr Probl Diagn Radiol 2018; 47 (01) 3-5
• 18 Shaw SJ, Jacobs B, Stockwell DC, Futterman C, Spaeder MC. Effect of a real-time pediatric ICU safety bundle dashboard on quality improvement measures. Jt Comm J Qual Patient Saf 2015; 41 (09) 414-420
• 19 James SH, Kimberlin DW. Neonatal herpes simplex virus infection. Infect Dis Clin North Am 2015; 29 (03) 391-400
• 20 Koopman RJ, Kochendorfer KM, Moore JL. , et al. A diabetes dashboard and physician efficiency and accuracy in accessing data needed for high-quality diabetes care. Ann Fam Med 2011; 9 (05) 398-405
• 21 Fox LA, Walsh KE, Schainker EG. The creation of a pediatric hospital medicine dashboard: performance assessment for improvement. Hosp Pediatr 2016; 6 (07) 412-419
• 22 Pageler NM, Longhurst CA, Wood M. , et al. Use of electronic medical record-enhanced checklist and electronic dashboard to decrease CLABSIs. Pediatrics 2014; 133 (03) e738-e746
• 23 Mussman GM, Conway PH. Pediatric hospitalist systems versus traditional models of care: effect on quality and cost outcomes. J Hosp Med 2012; 7 (04) 350-357
• 24 Hain PD, Daru J, Robbins E. , et al. A proposed dashboard for pediatric hospital medicine groups. Hosp Pediatr 2012; 2 (02) 59-68
• 25 Karami M, Langarizadeh M, Fatehi M. Evaluation of effective dashboards: key concepts and criteria. Open Med Inform J 2017; 11 (01) 52-57
• 26 Simpao AF, Ahumada LM, Larru Martinez B. , et al. Design and implementation of a visual analytics electronic antibiogram within an electronic health record system at a tertiary pediatric hospital. Appl Clin Inform 2018; 9 (01) 37-45
• 27 Wanderer JP, Gruss CL, Ehrenfeld JM. Using visual analytics to determine the utilization of preoperative anesthesia assessments. Appl Clin Inform 2015; 6 (04) 629-637
• 28 Caban JJ, Gotz D. Visual analytics in healthcare--opportunities and research challenges. J Am Med Inform Assoc 2015; 22 (02) 260-262
• 29 Thomas JJ, Cook KA. A visual analytics agenda. IEEE Comput Graph Appl 2006; 26 (01) 10-13
• 30 Ratwani RM, Fong A. ‘Connecting the dots’: leveraging visual analytics to make sense of patient safety event reports. J Am Med Inform Assoc 2015; 22 (02) 312-317
• 31 Radhakrishnan K, Monsen KA, Bae S-H, Zhang W. ; Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
• 32 Cerner. Available at: https://www.cerner.com/ . Accessed May 29, 2018
• 33 Qlik. Available at: www.qlik.com . Accessed January 1, 2018
• 34 Parikh K, Hall M, Mittal V. , et al. Establishing benchmarks for the hospitalized care of children with asthma, bronchiolitis, and pneumonia. Pediatrics 2014; 134 (03) 555-562
• 35 Feudtner C, Feinstein JA, Zhong W, Hall M, Dai D. Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr 2014; 14: 199
• 36 Stata Statistical Software. Release 13. Available at: https://www.stata.com/support/faqs/resources/citing-software-documentation-faqs/ . Accessed January 31, 2019
• 37 Perron CE, Bachur RG, Stack AM. Development, implementation, and use of an emergency physician performance dashboard. Clin Pediatr Emerg Med 2017; 18 (02) 115-123
• 38 Ralston SL, Atwood EC, Garber MD, Holmes AV. What works to reduce unnecessary care for bronchiolitis? A qualitative analysis of a national collaborative. Acad Pediatr 2017; 17 (02) 198-204
• 39 Venkatesh V, Bala H. Technology acceptance model 3 and a research agenda on interventions. Decis Sci 2008; 39 (02) 273-315
• 40 Rahimi B, Nadri H, Lotfnezhad Afshar H, Timpka T. A systematic review of the technology acceptance model in health informatics. Appl Clin Inform 2018; 9 (03) 604-634
• 41 Langley G, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass; 2009
• 42 Genies MC, Kim JM, Pyclik K, Rossi S, Spicyn N, Serwint JR. Impact of an educational intervention to improve physician adherence to bronchiolitis clinical practice guidelines: a pre-post intervention study. Clin Pediatr (Phila) 2018; 57 (03) 253-258
• 43 McCulloh RJ, Smitherman S, Adelsky S. , et al. Hospitalist and nonhospitalist adherence to evidence-based quality metrics for bronchiolitis. Hosp Pediatr 2012; 2 (01) 19-25
• 44 Cabana MD, Rand CS, Powe NR. , et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282 (15) 1458-1465
• 45 Casey Jr DE. Why don't physicians (and patients) consistently follow clinical practice guidelines?. JAMA Intern Med 2013; 173 (17) 1581-1583
• 46 Bryan MA, Desai AD, Wilson L, Wright DR, Mangione-Smith R. Association of bronchiolitis clinical pathway adherence with length of stay and costs. Pediatrics 2017; 139 (03) e20163432
• 47 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518