Design and Implementation of a Pediatric ICU Acuity Scoring Tool as Clinical Decision Support

Abstract

Background and Objective Pediatric in-hospital cardiac arrest most commonly occurs in the pediatric intensive care unit (PICU) and is frequently preceded by early warning signs of clinical deterioration. In this study, we describe the implementation and evaluation of criteria to identify high-risk patients from a paper-based checklist into a clinical decision support (CDS) tool in the electronic health record (EHR).

Materials and Methods The validated paper-based tool was first adapted by PICU clinicians and clinical informaticians and then integrated into clinical workflow following best practices for CDS design. A vendor-based rule engine was utilized. Littenberg's assessment framework helped guide the overall evaluation. Preliminary testing took place in EHR development environments with more rigorous evaluation, testing, and feedback completed in the live production environment. To verify data quality of the CDS rule engine, a retrospective Structured Query Language (SQL) data query was also created. As a process metric, preparedness was measured in pre- and postimplementation surveys.

Results The system was deployed, evaluating approximately 340 unique patients monthly across 4 clinical teams. The verification against retrospective SQL of 15-minute intervals over a 30-day period revealed no missing triggered intervals and demonstrated 99.3% concordance of positive triggers. Preparedness showed improvements across multiple domains to our a priori goal of 90%.

Conclusion We describe the successful adaptation and implementation of a real-time CDS tool to identify PICU patients at risk of deterioration. Prospective multicenter evaluation of the tool's effectiveness on clinical outcomes is necessary before broader implementation can be recommended.

Keywords

decision support systems - cardiopulmonary arrest - pediatrics

Background and Significance

Each year, more than 6,000 children in the United States experience in-hospital cardiac arrest with more than 90% of these events occurring in a pediatric intensive care unit (PICU).[1] [2] These events are frequently preceded by early warning signs of clinical deterioration but can easily be missed by busy bedside providers.[3] The implementation of early warning scores[4] [5] [6] [7] [8] [9] [10] [11] and rapid response teams[12] [13] [14] on general pediatric wards have led to improved recognition of clinical deterioration outside the PICU. Despite these advances, there has been little work to standardize prediction of deterioration events within the PICU. Therefore, there is little knowledge on the ability to identify, improve provider preparedness for, and ultimately intervene to improve the clinical course for the patients at greatest risk for cardiac arrest in the PICU. Previous studies have demonstrated that intense just-in-time training can improve survival outcomes for children who suffer from in-hospital cardiac arrest.[15] [16] [17] [18] To improve effectiveness and efficiency of just-in-time education and training for these high-risk patients, there is a clear need for standardized, objective methods to reliably identify patients at high-risk for clinical deterioration.

Prior to this work, the PICU at Children's Hospital of Philadelphia (CHOP) did not have a standardized way to identify patients at high risk for a clinical deterioration event, defined as a cardiac arrest or code bell activation. A process of twice daily safety huddles was in place, but the identification of patients was subjective—highly variable and dependent on the experience and confidence of the supervising attending. In response to this problem, a group of clinicians at CHOP developed criteria through expert consensus to help identify these patients in a more reliable manner.[19] This paper checklist, comprised of laboratory, physiological, and intervention-based criteria, performed well prospectively over 3 months (4,832 patient-days) with sensitivity and specificity both exceeding 95%. Although highly reliable, the paper checklist is not routinely used in clinical care due to the significant time commitment required to review each record and manually screen for the 15 criteria. Partnering with clinical informatics and quality improvement experts in the hospital, a project team was formed to develop and implement a real-time clinical decision support (CDS) tool, the PICU Warning Tool, based on the previously determined criteria. The aim of this effort was to describe the implementation and evaluation of criteria to identify high-risk patients from a paper-based checklist into a CDS tool in the electronic health record (EHR). This paper reports findings on scientific plausibility, technical feasibility, and preliminary process outcomes of the CDS tool, with future planned on outcome measures.

Materials and Methods

Setting

The study PICU is a tertiary care pediatric medical/surgical ICU with 55 beds and averages over 3,500 admissions per year in each of the past 5 years. The cardiac surgical ICU and neonatal ICU are separate from the PICU and did not take part in this quality improvement initiative.

Development: Determination of Prediction Criteria and Adaptation of Existing Paper Checklist

The process began with a review of the published checklist by PICU clinicians and clinical informaticians to determine if they were amenable to translation into a real-time CDS system.[19] Some criteria were excluded because the data were not reliably recorded as discrete data, or were not available in real time (e.g., procedure documentation several days after it was performed). Other criteria could not be included due to lack of a machine readable format, such as “provider intuition.” While documentation fields could certainly be added to the EHR to capture these missing elements, the team chose to adapt the set of checklist criteria and build a detection system without requiring any additional clinical documentation to best meet the objective of integration into existing workflows. Based on the limitations above, five criteria from the initial paper-based tool were excluded from the design of the CDS tool ([Table 1]).[19] The team elected to add criteria for severe cardiac dysfunction and electrolyte abnormalities that can impact cardiac contractility and arrhythmias since the tool was unable to capture the original cardiac criteria. In addition, due to the large increase in the use of noninvasive ventilation from the time of the original development of the checklist to implementation of the PICU Warning Tool, a criterion for high-risk patients on noninvasive ventilation was added. Given the extent of the modifications, the team acknowledged that the extent of modifications required to translate the paper-based tool represented substantial instrument alteration and that the test characteristics of the paper-based tool may not be directly extrapolated to the new electronic tool. Nonetheless, they felt it was important to proceed given the potential contribution to patient care this real-time tool could provide.

Table 1 Criteria from retrospective study and definition or rationale for exclusion

Criterion	Definition or reason for exclusion
Unchanged
Mean airway pressure > 20 cm H2O	Mean airway pressure more than 20 cm H2O for 2 h
ECMO	All ECMO (up to 24 h past decannulation)
Adapted
Nitric oxide therapy > 5 ppm	Most recent nitric oxide gas value > 5 ppm AND • Active pulmonary hypertension on problem list OR • Received medication indicating pulmonary hypertension in prior 24 h
Use of two vasoactive medications OR high dose single drug	MAR administrations within the last 6 h of: • Any dose of phenylephrine, dobutamine, or vasopressin • Any two of the following: dopamine, epinephrine, or norepinephrine • Epinephrine or norepinephrine with dose > 0.1 • Dopamine with dose > 5
Traumatic brain injury with intracranial hypertension	Sustained elevated intracranial pressure > 20 for most recent 2 h
PICU initiated renal replacement therapy	First 5 h on CRRT and first 5 h off CRRT
Potassium > 7.0	Two consecutive potassium laboratory results > 7 (resulted within the last 24 h)
pH < 7.10	Most recent pH laboratory value (resulted within the last 24 h) < 7.1
Lactate > 10	Lactate laboratory result > 10 without two consecutive lactate laboratory results < 10 (resulted within the last 24 h)
Arrhythmia	Any administration of medications indicating hemodynamically significant arrhythmia within the last 24 h: lidocaine IV, amiodarone IV, and adenosine IV
Excluded
Stage I hypoplastic left heart repair in past 24 h	Excluded. Operative procedures not discretely captured until days after procedure. Hypoplastic left heart patients not cared for in this PICU
Recent life-threatening event requiring code team activation or cardiac arrest	Excluded. No standardized documentation of these events
Intubation/extubation of known difficult airway	Excluded. Difficult airway is reliably captured in problem list by airway team, but nearly impossible to know when an intubation is upcoming or planned
Intubation of known diagnosis of myocarditis	Excluded. Myocarditis not reliably captured. Nearly impossible to know when an intubation is upcoming or planned, which is the time when notification is needed
Provider Intuition (e.g., patient too unstable for daily care)	Excluded. This is not captured currently
Added
Magnesium < 1	Most recent magnesium laboratory value (resulted within the last 24 h) < 1
Ionized calcium < 0.9	Most recent ionized calcium laboratory value (resulted within the last 24 hours) < 0.9
Noninvasive ventilation with high FiO2 requirement	Definition: FiO2 > 80 on positive pressure ventilation for most recent 2 h
Severe cardiac dysfunction	Mixed venous saturation < 60 without 2 consecutive values > 70 within the past 24 h OR milrinone infusion given within the previous 6 h

Abbreviations: CRRT, continuous renal replacement therapy; ECMO, extracorporeal membrane oxygenation; FiO₂, fraction of inspired oxygen; MAR, Medication Administration Record; PICU, pediatric intensive care unit.

Development: CDS User Goals

CDS user goals were generated by the multidisciplinary team leading the project, including critical care physicians, clinical informaticians, critical care nurses, respiratory therapists, and an improvement advisor. Given the low incidence of clinical deterioration events (baseline 4.8 events per 1,000 patient days), the system was expected to have a low positive predictive value (i.e., despite the high specificity, few identified as at-risk would have deterioration events). With this challenge in mind, the primary CDS goal identified by clinical providers was the improvement in situational awareness at the unit level. Situational awareness is the ability to recognize cues that increase the awareness of what is happening around you, integrate information to develop a comprehensive picture of the current state, and extrapolate forward to determine if the knowledge obtained will influence the situation immediately or in the near future.[20] [21] At the same time, given the findings in prior work, providers did want to identify individual patients at high risk to optimize bedside preparedness in the event of cardiac arrest. Lastly, providers needed to understand why a patient was thought to be at risk for deterioration (i.e., what at-risk criteria were met by the patient) for prompt and appropriate response.

Implementation: CDS Rule Engine

Based upon the risk criteria and workflow requirements, we considered multiple options for rule evaluation systems. The vendor EHR at CHOP, Epic Systems (Verona, Wisconsin, United States), has a rule evaluation feature that allows for a “scoring” system to evaluate multiple rules and a separate feature that allows for storing of the result using Boolean logic. Given the separation of these two rule engine functions, the vendor EHR's native rule engine required fairly complex build to handle the decision logic. Alternatively, the EHR also enables a Web services approach that would simplify the build for rule evaluation. The team ultimately chose to use the native vendor rule evaluation system despite the complex build because (1) at that time, there was no established institutional policy to allow transmission of protected health information to a CDS Web service and (2) the inpatient EHR team had limited experience with build outside the vendor EHR and could not support a Web service architecture with existing resources. Since the vendor tool did not automatically record scores in the record, a silent background process was configured to create an alert record whenever a patient was identified as meeting at least one at-risk criterion.

Implementation: Workflow and Interface

The goal of our workflow integration was to align with the CDS goals described above. For the unit situational awareness goal, an interruptive alert would have been quite burdensome, given the low prevalence of deterioration events.[22] In critical care settings, patient data review and prioritizing multiple patients are important clinician goals for situational awareness.[23] This team pursued a similar design, with a high level patient list display that allowed “drilling down” to more detailed individual patient information.

This drilling down capability would allow us to meet the second goal of providing details about why the patient was at risk. We implemented such an interface, with a high-level patient list column that triggered at-risk patients with color and icon indicators ([Fig. 1C]). Double-clicking or hovering over the column for any given patient provided a pop-up with details regarding which of the criteria had been met ([Fig. 1A], [B]), supporting the user need to identify why a patient was deemed at-risk. To further support this goal a notification banner was also displayed within the individual patient chart ([Fig. 1D]). The color choices aligned with standards from our hospital CDS committee, which had, in turn, based its standards on best practices from the clinical informatics literature.[24]

Fig. 1 (A, B) Detailed view of single and multiple criteria. (C, D) Patient list and banner views, on-demand detailed view.

To further support situational awareness, we also displayed information about at-risk patients on the wall-mounted widescreen displays used during safety huddles, shown in [Fig. 2A], [B]. Again, clicking the high-level display opened a display of the detailed clinical criteria.

Fig. 2 (A) Acuity system display for wall-mounted widescreen unit displays, unit view. (B) Acuity system display for wall-mounted widescreen unit displays, patient view.

Implementation: Iterative Testing, Feedback, and Revision

Given the complexity of documentation and interfaces with laboratory systems, the testing of this complex set of rules in a nonproduction environment was limited, which created the risk of both technical and usability issues.[25] To address these risks, the system was tested in a multistep process: (1) use case-based testing of rule logic in a development environment with the production EHR's user interface to ensure that the CDS system was producing the expected results, (2) limited initial implementation in the production system with access instructions provided to clinicians on the project team only, and, finally, (3) full implementation in the production system with an embedded clinician feedback survey and ongoing postimplementation monitoring of the system.

To perform initial verification by clinicians, the team created an easily completed survey form using REDCap (Vanderbilt University),[26] the hospital-supported survey tool. More active approaches like user interviews were not practical as they were expected to disrupt patient care. A hyperlink to the survey, which contained fields for discrete capture of correct or incorrect criteria and optional fields for comments, was embedded in the EHR workflow. In addition to the embedded survey, ongoing postimplementation surveillance relied on the comparison of patients identified by the EHR as being at-risk compared with a retrospective query of our data warehouse for patients who met the same criteria (see the “Evaluation” section). Testing and implementation of the first set of criteria took place in November 2015, and all criteria were active in the production system for clinical use by April 2016.

Evaluation

For evaluation of this tool, we used the Littenberg Technology Assessment in Medicine framework,[27] which centers on five levels. Based on this methodology, the project completed evaluation based on levels 1 to 3, as outlined in [Table 2].

Table 2 Littenberg framework-based evaluation questions applied to our CDS tool

Assessment	Project questions	Outcome of interest
1. Plausibility	• Does it make sense to build?	- Consensus among clinicians
2. Technical feasibility	• Are the results correct? • Does it work as designed?	- Comparison of system firing against retrospective SQL-based query of patients meeting criteria - User reported issues in embedded survey
3. Intermediate/Process outcomes	• Did preparedness improve?	Preparedness survey
4. Patient outcomes	• Did it impact clinical event rates?	- Rate of clinical deterioration events
5. Societal outcomes	• Can it scale to other environments?	- Decreased morbidity and mortality in hospitalized pediatric patients

Abbreviations: CDS, clinical decision support; SQL, Structured Query Language.

Plausibility of the tool was adjudicated based on PICU clinician consensus building on the established value of the initial paper-based tool.[19] Evaluation of technical feasibility entailed several approaches. Data verification was initially evaluated using the aforementioned integrated feedback survey (see [Appendix A]). A more comprehensive verification of the CDS rule engine's data quality and accuracy was completed by retrospectively verifying the at-risk patients identified by the CDS system against a retrospective Structured Query Language (SQL) query of the institutional data warehouse based on the same criteria. Although labor-intensive to recreate, the purpose of this query was twofold. First, the code driving the rules for the CDS rule engine was not transparent in the vendor EHR. A retrospective query of the data warehouse for patients who met the at-risk criteria would therefore identify both incorrectly triggered and missed patients in comparison to the real-time tool. Second, a retrospective query was the only way to evaluate the preintervention period before the tool was live in the EHR.

There were some challenges in setting up this comparison, which at their core relate to the different database structures used in our EHR. The frontend database, where the CDS rule engine operates, is a hierarchical transactional event-driven database. The SQL verification was completed using a second relational database that is updated on a daily basis with data from the hierarchical database. The background alert tracking the CDS system was transactional, only getting triggered when a user or process (e.g., interface) interacted with the system in some way (opening chart, recording data). A retrospective query of the relational database did not have this limitation since it could detect the time of the original data element and therefore could much more precisely identify the moment of a flag. Given patient acuity in the PICU, interactions with the chart are quite frequent, so there was never a concern that this was resulting in delays in the appearance of triggers during patient care. The implication for the verification, however, was that to reconcile these two database structures we needed to consider a time window to serve as a proxy for an event in the EHR, essentially a sampling frequency. We initially considered a 1-hour interval, to align with the frequency of the background alert in the real-time database, but opted for a more granular 15-minute sampling interval since the clinicians felt that the smaller interval more accurately reflected how often charts were opened. This would give us more granular data for comprehensive review, but for the purposes of verification we did allow for 2-hour windows to count as a match. In short, we created a binary comparison script that evaluated 15-minute intervals and if a trigger was detected within 2 hours of the real-time background alert it was counted as a match.

Intermediate/process outcomes are important in this work as the patient outcome, clinical deterioration event, is rare. Our process improvement goal focused on improvement in bedside preparedness for high-risk patients, determined by a survey of bedside providers on nonholiday week days. This tool (see [Appendix B]) was developed by the multidisciplinary research team after examination of the key drivers of clinical deterioration. The survey was administered to bedside providers including bedside nurses, respiratory therapists, and resident physicians (or nurse practitioners) following rounds on nonholiday weekdays at the convenience of our study team. The survey tool assessed the accurate identification of high-risk patients and development of a mitigation plan that addressed the domains identified by the research team—situational awareness/teamwork (comprehension of risk), education/resources, assessment, environment, and medications/equipment. Scoring was completed by a member of the study team in consultation with patient's attending physician.

Results

Implementation and Verification (Technical Feasibility)

The system began verification in the live production system in November of 2015 and went live for clinical use in April of 2016 (7,718 patient days). At the time of manuscript submission, it continues to be used operationally. The system evaluates approximately 340 unique patients monthly across 4 clinical teams in the PICU.

The verification and technical feasibility process began with 315 user-initiated feedback forms that were completed by clinicians across 182 unique patients. Most (90%) were submitted during the period of background verification before broad clinical implementation, for initial analysis of the technical feasibility and usability/workflow integration by the research team. During this stage of background testing in the production environment, the surveys identified issues in rule configuration via the embedded survey that were not identified during testing in nonproduction environments. In total, the survey reviewed 2,176 criteria with 59 (3%) containing either false positive or negative alert failures. A comparison with SQL queries was also done during this preliminary phase, with additional alert failures identified. Each of the failures was investigated and addressed in the underlying rule engine. Primarily these issues were related to unanticipated rule failures (e.g., text results for laboratories were interpreted as 0 and infinity when a numerical value was expected by the rule engine, system did not trigger on laboratory result entry in the production environment but did so in the development environment). Introduction of the tool did not impact the overall EHR system performance, although the task of loading a patient list with the embedded tool could take several seconds longer than usual when several units were combined in a single list of more than 50 patients. Viewing that many patients at a single time was uncommon, however, and clinicians did not report this as an issue.

After this iterative refinement process, a comprehensive verification of system accuracy with retrospective queries of 15-minute intervals over a 30-day period revealed no missing triggered intervals and 99.3% of positive triggers within the real-time system were corroborated in the SQL database. The remaining 0.7% were due to differences in how the criteria were defined in the real-time versus retrospective code design as described in the “Materials and Methods” section. In the majority of these infrequent cases, the retrospective query missed the leading or lagging end of a patient's triggered interval. In addition, we observed a small number of triggers in the real-time system due to documentation or results that were corrected at a later time by a clinician or interface and were therefore not detected by the SQL query.

Intermediate/Process Outcomes

Preparedness results, as measured in baseline, of pre-PICU Warning Tool implementation and post-PICU Warning Tool implementation surveys showed improvements across multiple domains to our a priori goal of 90% as shown in [Fig. 3]. Baseline preparedness surveys (N = 34) showed overall preparedness at 74% with component scores of situational awareness at 62%, medications/equipment at 76%, assessment at 66%, education/resources at 97%, and environment at 70%. Quality improvement interventions including standardized twice daily nursing and respiratory therapy huddles in addition to baseline twice daily multidisciplinary huddles, standardization of transports outside the PICU, and adapting the format of the twice daily safety huddles led to improvements in preimplementation survey (N = 61) preparedness to 84% over the 3 months prior to PICU Warning Tool implementation. Postimplementation surveys (N = 42) showed further improvement to 94% overall with situational awareness at 90%, medications/equipment at 94%, assessment at 90%, education/resources at 99%, and environment at 98%. Components were also evaluated by bedside provider role and demonstrated improvement across roles with postimplementation preparedness for bedside nurses (N = 19) at 92%, respiratory therapists (N = 16) at 95%, and resident physicians/nurse practitioners (N = 7) at 98%.

Fig. 3 Preparedness assessed via simple survey of frontline care providers including nursing, physicians, and respiratory therapists demonstrated improvement of ∼90% following implementation of the clinical decision support tool.

Discussion

This study describes the adaptation of a paper-based checklist into an automated CDS tool, the PICU Warning Tool, to improve shared situational awareness. With a SQL verification process, we determined the tool performs with high reliability. Deployment of the tool was associated with improvements in situational awareness across several key domains and further patient outcomes remain under study.

This development and application of a CDS system to predict high-risk PICU patients is unique in its clinical content and application of an evaluation framework. Automated systems to identify patients at risk for deterioration have long been a goal for health care and many have shown promise in research trials.[28] [29] [30] [31] Yet, widespread adoption of these systems has lagged due to high false-alarm rates, lack of impact on patient outcomes, and poor CDS design. Early warning scores, which share many features in common with this PICU Warning Tool, are focused on improving detection of clinically deteriorating patients in lower acuity settings: outside of ICUs on regular medical surgical wards.[4] [5] [6] [7] Despite being broadly implemented throughout the United States and much of the developed world, evidence supporting pediatric early warning score use is limited. A recent 21-hospital cluster-randomized trial showed no difference in mortality with implementation of the Bedside Pediatric Early Warning System, although this may have been due in part to lower baseline mortality rates than anticipated.[32] More advanced algorithms demonstrate better rates of prediction of deterioration, yet these tools have not been validated for the PICU population.[31] [33] In addition, the PICU at CHOP wanted to identify a high-risk subset of deteriorating patients—those at risk for cardiac arrest or code bell activation.

Another unique feature of this CDS evaluation is the application of an evaluation framework, which notably included verification of the system against a retrospective query of patient data. Retrospective verification with a second data source is not typically done with rule-based CDS implementations,[34] but we recommend a verification process for CDS that will be used in high acuity settings or is of significant complexity. On a related note, consistent with prior literature,[25] we were able to identify multiple unanticipated issues by incorporating a testing phase in the production environment and would endorse this approach whenever possible.

The most significant limitations in this work relate to its scalability, which in turn centers on the build challenges of this CDS tool. The predominant theme of the challenges was the relatively poor ability of native EHR rule engine to recognize anything but the simplest of patterns. As various workarounds were considered, the informaticians discussed the relative advantages and disadvantages of building the tool using the vendor-supplied rule engine versus a vendor-neutral approach that could integrate using a standard like Health Level-7 Fast Healthcare Interoperability Resources. The team considered relative advantages and disadvantages of these approaches, and a summary of some of the key considerations is reviewed in [Table 3].

Table 3 Advantages and disadvantages in vendor versus nonvendor approaches

	Advantages	Disadvantages
Vendor-based	• Uses existing IS skills and support model • Fast and reliable performance • Many options for workflow integration	• Constraints in rule-engine design • Code updates by vendor or in local data structure can break rules • Not scalable to other hospitals, even those on same vendor would need to rebuild for local data
Vendor-agnostic	• Greater control and flexibility in rule engine design • More easily scalable to other hospitals, although data elements from areas without interoperability standards may still require local mapping	• Significant resource investment to design and support • Potential performance or reliability problems

Another limitation, one that applies to the majority of CDS systems, is the risk of malfunction over time due to changes in underlying data structure or vendor code updates. Although this tool will be monitored with our current infrastructure, which includes a CDS committee and a dashboard to track activity of alerts and other CDS interventions, it is worth noting that this monitoring requires active human surveillance. In addition, it will only show significant deviations in performance of the overall system, not malfunction of individual decision rules.

A final limitation relates to the previously described significant modifications made when compared with the paper-based checklist. As patient outcomes are studied in future work, should the sensitivity of the PICU Warning Tool prove inadequate due to the lack of these criteria, changes to documentation workflows would be considered to incorporate some of the more subjective criteria. While this would improve the sensitivity of the tool, it would come at the cost of being dependent on additional time and documentation by clinicians.

Conclusion

This article describes the design, data integrity assurance, and implementation of a new CDS tool for detecting deterioration in the pediatric intensive care setting. The process reinforced many of the known challenges in translating clinical rules and guidelines into machine executable code. We were able to overcome the inherent limitations of risk scores with low positive predictive value and the vendor-based rule engine by following best practices for CDS design, testing, and evaluation. We are looking forward to additional verification and iterative improvement of the tool so that patient outcome measures can be analyzed in the future.

Clinical Relevance Statement

Implementation of a new Clinical Decision Support system can be challenging, and partnership with quality improvement teams and being systemic in the approach are key factors for success. Verification in a live production system before full deployment should be considered, especially when complex interfaces are involved in the CDS rule criteria. Replicating CDS rule engine build with a retrospective data query can be of great value to evaluate the performance of the CDS and provide insights into the preintervention period.

Multiple Choice Question

Which statistical measure is the most helpful in determining how interruptive EHR-based Clinical Decision Support should be?

• Sensitivity

• Specificity

• Positive predictive value (i.e., true positive rate)

• Negative predicative value (i.e., true negative rate)

• Likelihood ratio

Correct Answer: The correct answer is option c, positive predictive value. Several factors can determine the format of a CDS intervention, but of the statistical measures positive predictive value (true positives/all positives) is most useful in determining how interruptive and prescriptive CDS should be.

Conflict of Interest

None.

Acknowledgments

The authors are grateful to Ron Keren and the Office of Quality Improvement at The Children's Hospital of Philadelphia for their support of this effort.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was approved by the Institutional Review Board of CHOP with a waiver of consent.

• References

• 1 Berg RA, Sutton RM, Holubkov R. , et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network and for the American Heart Association's Get With the Guidelines-Resuscitation (formerly the National Registry of Cardiopulmonary Resuscitation) Investigators. Ratio of PICU versus ward cardiopulmonary resuscitation events is increasing. Crit Care Med 2013; 41 (10) 2292-2297
• 2 Knudson JD, Neish SR, Cabrera AG. , et al. Prevalence and outcomes of pediatric in-hospital cardiopulmonary resuscitation in the United States: an analysis of the Kids' Inpatient Database. Crit Care Med 2012; 40 (11) 2940-2944
• 3 Morrison LJ, Neumar RW, Zimmerman JL. , et al; American Heart Association Emergency Cardiovascular Care Committee, Council on Cardiopulmonary, Critical Care, Perioperative and Resuscitation, Council on Cardiovascular and Stroke Nursing, Council on Clinical Cardiology, and Council on P. Strategies for improving survival after in-hospital cardiac arrest in the United States: 2013 consensus recommendations: a consensus statement from the American Heart Association. Circulation 2013; 127 (14) 1538-1563
• 4 Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care 2009; 13 (04) R135
• 5 Bonafide CP, Holmes JH, Nadkarni VM, Lin R, Landis JR, Keren R. Development of a score to predict clinical deterioration in hospitalized children. J Hosp Med 2012; 7 (04) 345-349
• 6 Duncan H, Hutchison J, Parshuram CS. The Pediatric Early Warning System score: a severity of illness score to predict urgent medical need in hospitalized children. J Crit Care 2006; 21 (03) 271-278
• 7 Parshuram CS, Duncan HP, Joffe AR. , et al. Multicentre validation of the bedside paediatric early warning system score: a severity of illness score to detect evolving critical illness in hospitalised children. Crit Care 2011; 15 (04) R184
• 8 Haines C, Perrott M, Weir P. Promoting care for acutely ill children-development and evaluation of a paediatric early warning tool. Intensive Crit Care Nurs 2006; 22 (02) 73-81
• 9 Tucker KM, Brewer TL, Baker RB, Demeritt B, Vossmeyer MT. Prospective evaluation of a pediatric inpatient early warning scoring system. J Spec Pediatr Nurs 2009; 14 (02) 79-85
• 10 Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score. Crit Care Med 1996; 24 (05) 743-752
• 11 Edwards ED, Powell CVE, Mason BW, Oliver A. Prospective cohort study to test the predictability of the Cardiff and Vale paediatric early warning system. Arch Dis Child 2009; 94 (08) 602-606
• 12 Sharek PJ, Parast LM, Leong K. , et al. Effect of a rapid response team on hospital-wide mortality and code rates outside the ICU in a Children's Hospital. JAMA 2007; 298 (19) 2267-2274
• 13 DeVita MA, Smith GB, Adam SK. , et al. “Identifying the hospitalised patient in crisis”—a consensus conference on the afferent limb of rapid response systems. Resuscitation 2010; 81 (04) 375-382
• 14 Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of rapid response system implementation on critical deterioration events in children. JAMA Pediatr 2014; 168 (01) 25-33
• 15 Sutton RM, Niles D, Meaney PA. , et al. Low-dose, high-frequency CPR training improves skill retention of in-hospital pediatric providers. Pediatrics 2011; 128 (01) e145-e151
• 16 Niles D, Sutton RM, Donoghue A. , et al. “Rolling Refreshers”: a novel approach to maintain CPR psychomotor skill competence. Resuscitation 2009; 80 (08) 909-912
• 17 Sutton RM, Niles D, Meaney PA. , et al. “Booster” training: evaluation of instructor-led bedside cardiopulmonary resuscitation skill training and automated corrective feedback to improve cardiopulmonary resuscitation compliance of Pediatric Basic Life Support providers during simulated cardiac arrest. Pediatr Crit Care Med 2011; 12 (03) e116-e121
• 18 Wolfe H, Zebuhr C, Topjian AA. , et al. Interdisciplinary ICU cardiac arrest debriefing improves survival outcomes*. Crit Care Med 2014; 42 (07) 1688-1695
• 19 Niles DE, Dewan M, Zebuhr C. , et al. A pragmatic checklist to identify pediatric ICU patients at risk for cardiac arrest or code bell activation. Resuscitation 2016; 99: 33-37
• 20 Goldenhar LM, Brady PW, Sutcliffe KM, Muething SE. Huddling for high reliability and situation awareness. BMJ Qual Saf 2013; 22 (11) 899-906
• 21 Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors J Hum Factors Ergon Soc 1995; 37: 65-84
• 22 Romero-Brufau S, Huddleston JM, Naessens JM. , et al. Widely used track and trigger scores: are they ready for automation in practice?. Resuscitation 2014; 85 (04) 549-552
• 23 Wright MC, Dunbar S, Macpherson BC. , et al. Toward designing information display to support critical care. A qualitative contextual evaluation and visioning effort. Appl Clin Inform 2016; 7 (04) 912-929
• 24 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 25 Wright A, Aaron S, Sittig DF. Testing electronic health records in the “production” environment: an essential step in the journey to a safe and effective health care system. J Am Med Inform Assoc 2017; 24 (01) 188-192
• 26 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 27 Littenberg B. Technology assessment in medicine. Acad Med 1992; 67 (07) 424-428
• 28 Rothman MJ, Tepas III JJ, Nowalk AJ. , et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform 2017; 66: 180-193
• 29 Escobar GJ, Turk BJ, Ragins A. , et al. Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals. J Hosp Med 2016; 11 (Suppl. 01) S18-S24
• 30 Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM 2001; 94 (10) 521-526
• 31 Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system. J Hosp Med 2014; 9 (02) 116-119
• 32 Parshuram CS, Dryden-Palmer K, Farrell C. , et al; Canadian Critical Care Trials Group and the EPOCH Investigators. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the epoch randomized clinical trial. JAMA 2018; 319 (10) 1002-1012
• 33 Kirkland LL, Malinchoc M, O'Byrne M. , et al. A clinical deterioration prediction tool for internal medicine patients. Am J Med Qual 2013; 28 (02) 135-142
• 34 Byrne CB, Sherry DS, Mercincavage L. , et al. Key Lessons in Clinical Decision Support Implementation. Westat Tech Report; 2010