Optimizing Clinical Monitoring Tools to Enhance Patient Review by Pharmacists

Abstract

Background The Clinical Monitoring List (CML) is a real-time scoring system and intervention tool used by Mayo Clinic pharmacists caring for hospitalized patients.

Objective The study aimed to describe the iterative development and implementation of pharmacist clinical monitoring tools within the electronic health record at a multicampus health system enterprise.

Methods Between October 2018 and January 2019, pharmacists across the enterprise were surveyed to determine opportunities and gaps in CML functionality. Responses were received from 39% (n = 162) of actively staffing inpatient pharmacists. Survey responses identified three main gaps in CML functionality: (1) the desire for automated checklists of tasks, (2) additional rule logic closely aligning with clinical practice guidelines, and (3) the ability to dismiss and defer rules. The failure mode and effect analysis were used to assess risk areas within the CML. To address identified gaps, two A/B testing pilots were undertaken. The first pilot analyzed the effect of updated CML rule logic on pharmacist satisfaction in the domains of automated checklists and guideline alignment. The second pilot assessed the utility of a Clinical Monitoring Navigator (CMN) functioning in conjunction with the CML to display rules with selections to dismiss or defer rules until a user-specified date. The CMN is a workspace to guide clinical end user workflows; permitting the review and actions to be completed within one screen using EHR functionality.

Results A total of 27 pharmacists across a broad range of practice specialties were selected for two separate two-week pilot tests. Upon pilot completion, participants were surveyed to assess the effect of updates on performance gaps.

Conclusion Findings from the enterprise-wide survey and A/B pilot tests were used to inform final build decisions and planned enterprise-wide updated CML and CMN launch. This project serves as an example of the utility of end-user feedback and pilot testing to inform project decisions, optimize usability, and streamline build activities.

Keywords

clinical information systems - clinical decision support - monitoring and surveillance - workflows and human interactions

Background and Significance

Adverse drug events during hospitalization pose a serious risk to patient safety and care outcomes. It is estimated that approximately 5% of patients experience an adverse drug event during a hospital stay, and each additional day of hospitalization increases the risk by 0.5 percent.[1] To reduce patient risk, clinical decision support (CDS) and computerized scoring systems have been supported as evidence-based methods to improve medication-related patient safety.[2] [3] [4] [5] [6]

Mayo Clinic recently underwent a staged implementation of a new EHR across its enterprise of 19 hospitals. This implementation has connected all departments of pharmacy together with a common tool for monitoring and intervening on medication and patient-related problems, referred to as the Clinical Monitoring List (CML). Following EHR and CML implementation, organizational leadership planned a 6-month post go-live clinical retreat to evaluate real-world use of clinical tools in the new EHR. The intent of this retreat was to identify which tools had opportunities for enhancement and scope build enhancements for optimization prioritization using pharmacist end-user feedback.

In preparation for the clinical retreat, the CML was reviewed. The purpose of this project was to identify gaps in CML functionality and develop solutions to address deficits to allow for expansion and improvement of pharmacist services. A secondary goal was to demonstrate the efficacy of bidirectional feedback between clinical end-users and project team informaticists to promote successful design and implementation of CDS.

Methods

Optimization of the CML involved a three-staged approach. This started with an enterprise-wide pharmacist survey to assess opportunities for functionality enhancement. Following gap assessment, pilot build was developed and A/B tested in an effort to provide feedback from hands-on use of build adjustments to tailor modifications to further suit end-user needs. The third stage involved implementation of the optimized monitoring tools across the enterprise.

Stage 1

Study Design and Setting

The gap-analysis occurred at all sites in the Mayo Clinic enterprise. These sites include flagship hospitals located in Rochester, Minnesota, Jacksonville, Florida and Phoenix, Arizona, as well as health-system hospital sites located across Southern Minnesota and Southwest Wisconsin. The enterprise-wide analysis permitted inclusion of the spectrum of institution types including academic medical centers, community hospitals, and critical access hospitals.

Pharmacist Selection

Pharmacists eligible to complete the gap-analysis survey were those that worked as an inpatient pharmacist as their primary work location at any of the Mayo Clinic enterprise hospital sites between November 8, 2018 and January 9, 2019. Pharmacist managers and informaticists were excluded; as their time spent performing clinical duties is significantly limited in comparison to pharmacists that engage in a primarily clinical role.

Data Collection and Definitions

Pharmacists eligible to complete the CML survey were notified of their eligibility through an email including a hyperlink to REDCap; the online survey site.[7] The survey instrument included questions directly relating to columns within the CML, as the structure of the list involves multiple columns with integrated scoring systems and rule logic coinciding with a systems-based or problems-based patient review. Questions assessed functionality in columns that were working well, recommendations for rule adjustments within columns, and an appraisal of elective columns that pharmacists had added to their list to enhance their workflows. To assess information not previously incorporated into the CML, but desired by pharmacists, questions relating to information monitored on all or most patients were included.

Stage 2

Study Design and Setting

Following assessment of survey results, an updated CML and new Clinical Monitoring Navigator (CMN) were designed and developed by the informaticist authors to address the functional requests from the pharmacist survey for A/B testing. A/B testing is a basic experiment to compare two versions of design to determine which version performs better according to user feedback.[8] To scope out appropriate build interventions, a failure mode and effects analysis was performed. All preexisting CML rules were grouped into categories based on rule type. For example, all laboratory monitoring rules were grouped together because potential failure modes were similar for all such rules. Following analysis by the informaticist authors, recommended actions were developed to incorporate in the build for A/B testing and were granted approval to build by enterprise inpatient clinical pharmacy managers. The A/B testing compared the preexisting CML with the new pilot CML and CMN.[9] [10] [11] To release the build for A/B testing, the CML and CMN were made accessible to all pharmacists, but not readily visible. Pharmacists engaged in the pilot were instructed on how to access the build components and favorite them, so the build would be their default view upon opening the EHR. Two sequential pilots were completed by a group of selected pharmacists. The pilots took place in sequence in accordance with waterfall project management methodology ([Fig. 1]).[8] The purpose of using this methodology was to promote adherence to timeline objectives, improve testing ease for the informaticists, and increase the likelihood that end-users could adopt and evaluate the new technology changes quickly. Pilot durations were selected to ensure that pharmacists had ample opportunity to use the CML and CMN, while recognizing that it was not sustainable to have pharmacists complete their required clinical responsibilities, and also trial additional and separate technology for an extended period of time. The first pilot took place between January 7, 2019 and January 18, 2019 and involved only the updated CML. The second pilot took place between January 28, 2019 and February 8, 2019 and incorporated a novel CMN working in conjunction with the updated CML rule logic and scoring systems to allow the dismissal and deferral of patient scoring rules ([Fig. 2A], [B]). The strategy and framework for this pilot were to build a prototype that provided the function and display of the CMN with minimal build effort.[12] To accomplish this, we selected 2 out of 116 preexisting CML rules to completely build out for visual display and technical functionality of the deferral capabilities.

Fig. 1 Discrete project steps. Consistent with waterfall project management methodology, clinician feedback was collected at the beginning of the project and a sequential project plan was developed in accordance with those requests. CML, Clinical Monitoring List; CMN, Clinical Monitoring Navigator.

Fig. 2 (A) Identifying active rules in the Clinical Monitoring List. Hovering over the icons in the Clinical Monitoring List permits the pharmacist to view the rules that are firing. Double clicking the icon opens the Clinical Monitoring Navigator section associated with the icon that was clicked. (B) Interacting with Rules in the Clinical Monitoring Navigator. Within the Clinical Monitoring Navigator, the pharmacist reviews pertinent clinical details. Once completed, the pharmacist defers the rule to a date and time of their choosing.

Pharmacist Selection

Inpatient pharmacists from all enterprise sites were eligible for participation in the pilot tests. Regional pharmacy managers selected pharmacists representing a diverse assortment of clinical specialties and practice sites. Large sites with multiple subspecialties selected one pharmacist from each subspecialty group. Smaller sites selected one to three pharmacists from their overall pharmacist pool for participation. We requested a minimum number of 20 total but had no upper limit maximum. The goal of this variety was to ensure the CML and CMN worked across specialties, as the display was the same for all pharmacists, with variation in rule firing based on patient characteristics.

Data Collection and Definitions

Prior to pilot initiation, the informatics pharmacists prepared and emailed detailed instructions to the pharmacists engaging in the pilot regarding how to set up their pilot CML and CMN. Pharmacists were encouraged to share potential break/fixes with the informatics team throughout the course of the pilots, so they could be addressed in real time. Midway through each pilot, the lead author prepared and emailed REDCap surveys to obtain feedback on pilot CMN or CML usability.[7] Questions for the first pilot involving the CML focused on column updates and rule build. Questions for the second pilot for the CMN focused on the functionality and display of rule dismissal and deferral logic within the navigator. Given the vast customization options for the CMN, we focused our pilot survey questions directly to user experience with rule display for dismissal and deferral.

Stage 3

Study Design and Setting

Following pilot testing, the updated CML and new CMN were implemented. Principles from project management and implementation science were used to support sustainable and scalable change.[13] [14] Prior to implementation, pharmacists across the enterprise were educated on the utilization of the updated CML and CMN through in-person training sessions. Communication and education of pharmacists started 2 months before the implementation to ensure adequate training and understanding of new tools. Pharmacist managers engaged as liaisons to address practice changes and daily work prioritization for end-users to communicate a clear message about expectations following the implementation. During the implementation, the project team was available for break/fixes through a web-based ticketing system.

Results

Baseline Data Collection

Of the 416 pharmacists eligible to complete the clinical monitoring survey, 162 provided responses for a 39% response rate. Primary themes identified through survey responses included the desire for automated checklists of tasks, additional rule logic with closer alignment to clinical practice guidelines, and the ability to dismiss and defer rules ([Table 1]). Items frequently requested for automated checklists included orders associated with pharmacy to dose consults including those for warfarin, direct thrombin inhibitors, and total parenteral nutrition. Multiple requests communicated a need for guideline-based kidney monitoring rules, especially rules aligning with current evidence-based criteria for the diagnosis and staging of acute kidney injury.[15] Rules with the most significant interest in dismiss and defer logic included laboratory result rules, intravenous to oral conversion recommendations, and therapeutic drug monitoring flags. Nuances in patient care identified through survey responses allowed scoring system rules to be updated in a way that limited inaccurate rule firing and optimized the clinical utility of the list. For example, the rule logic behind the icon used to notify pharmacists of an impending patient discharge relied solely on a specific ordering process. This process was not followed uniformly. Therefore, the rule logic was updated to ensure that pharmacists are notified of an impending discharge regardless of the discharge ordering process used.

Table 1 Baseline survey themes

Column name	Column contents	Enterprise survey themes
Medication reconciliation	Provider review of prior to admission medications status and MAR hold pharmacist review status	Flagging in this icon relates to provider workflows rather than pharmacy workflows. Recommend to remove and replace with pharmacy-centric rules.
Discharge medication reconciliation	Discharge or prepare to discharge order signed status, provider completion of discharge medication review	Prepare for discharge orders are not consistently signed early enough to allow time for medication review. Recommend to replace with icon to display when provider has completed their medication review, so pharmacists know when it is time to review the discharge medication list.
Anticoagulation	Scoring system displaying rules regarding monitoring or high-risk anticoagulants and status of pharmacist or provider to dose anticoagulant medications	High-risk drugs consulted to be dosed by pharmacists including warfarin, and direct thrombin inhibitors are easily overlooked because they are incorporated to one icon for all anticoagulation rules.
Infectious diseases	Scoring system displaying antimicrobial stewardship rules, active antimicrobials, and recent antimicrobial-related laboratory values	Inability to dismiss rules makes it challenging to know if stewardship review has been completed daily.
Nephrology	Scoring system displaying changing renal function rules and renal dosing rules	Flagging for changing renal function is too sensitive and flagging renal impairment in patients that have relatively good renal function.
Needs review	Scoring system displaying monitoring rules not directly related to anticoagulation, infectious diseases or nephrology. Rules include intravenous to oral conversions, therapeutic drug monitoring laboratory values, active consults, and high-risk drug monitoring.	Inability to dismiss rules makes it challenging to know if rule review has been completed daily.
Warfarin	Manual completion checklist of patients with active pharmacy to dose warfarin consults	Manual completion is not unchecked when a related order is discontinued appropriately or inappropriately during the course of the day.

Abbreviation: MAR, medication administration record.

Note: During the initial stage of the project, a survey was sent to all inpatient pharmacists within the enterprise to assess utility of column build and obtain recommendations for optimization updates and changes.

Survey results were used to design and execute updates to the CML. Rule update requests were evaluated within the scope of enterprise-wide clinical practice standards and integrated if recommendations aligned with practice goals. The overwhelming desire to integrate dismissal and deferral of rules identified in the survey was addressed through the development of a CMN for the second pilot.

Pilots

Pilot test 1 evaluated pharmacist response to Clinical Monitoring List rule updates. Twenty-one pharmacists were selected from practice sites and specialties across the enterprise for participation. Surveys administered to participants at the end of the pilot identified significantly positive response to rule updates. Minimal adjustments were recommended for additional update or revision, with most feedback expressing the continued desire for rule logic to incorporate dismissal and deferral logic. Notably, this pilot provided evidence for the utility of performing A/B testing in a variety of clinical specialties, as most rule logic revisions arose from nuances in the pediatric population. Survey feedback themes provided for each column are provided in [Table 2].

Table 2 Pilot 1 survey themes

Column	Pilot 1 changes	Pilot 1 survey themes
Medication reconciliation	Renamed “admission” Scoring systems removed and replaced with two unique icon indicators; one for admission medication history completion and another for admission medication reconciliation completion Updated icons dynamically update as work is completed to move from an hourglass to indicate a task not completed to a blank icon to indicate that no further admission tasks need to be completed	Icons are easily interpreted and queue admission task completion in a logical order
Discharge medication reconciliation	Renamed “discharge” Scoring systems removed and replaced with two unique icon indicators; one for notification of a pending discharge and another for pharmacist discharge medication reconciliation completion Updated indicators dynamically update as work is completed to move from an hourglass to indicate a task not completed to a check mark to indicate a task that has been completed	Dynamic icons provide accurate notification of discharge plans. Scoring system for medication reconciliation documents stepwise completion.
Anticoagulation	In addition to preexisting score display icon, established two separate icons for consults for pharmacy to dose warfarin and direct thrombin inhibitors to aid in identification	Separate icons for high-risk drugs effectively ensure that monitoring does not get missed
Infectious diseases	Restricted lookback duration to 24 h on out of range drug monitoring laboratories	Condensed lookback period ensures that all notifications are relevant for immediate action
Nephrology	Updated rule logic to follow KDIGO AKI guidelines When patient is on any form of dialysis, updated the icon display to change color from orange to blue	Alignment with AKI guidelines helps to identify patients that need close review Color change with dialysis is convenient to identify the need for closer dosing evaluation
Needs review	Restricted lookback duration to 24 h on out of range drug monitoring laboratories For drugs that are triggering the intravenous to oral conversion flag, added a display to show the specific medication order that the rule identified	Condensed lookback period ensures that all notifications are relevant for immediate action Medication display in intravenous to oral conversion flags shortens the time needed to evaluate conversion candidates
Warfarin	Removed and replaced with automated warfarin-specific icon in the anticoagulation column that dynamically changes to a check mark when a warfarin order is placed or administered for the day	Automatic icon removes manual step of documenting task completion and ensures that the medication has been ordered and remains active throughout the day

Abbreviation: KDIGO AKI, Kidney Disease: Improving Global Outcomes Clinical Practice guideline for acute kidney injury.

Note: Pilot 1 focused on build changes that affected the rules and column display of the pharmacist Clinical Monitoring List. Pharmacists engaging in the pilot were surveyed for additional feedback. Responses were overwhelmingly positive and reflected progression toward optimization goals.

Pilot test 2 evaluated pharmacist response to the newly developed CMN, introducing the ability to dismiss and defer rules displaying on the CML ([Fig. 2A], [B]). Feedback received from survey responses was generally positive. Pharmacists were pleased with the ability to dismiss and defer rules to declutter their views. Due to the purposefully incomplete build of the prototype CMN, pharmacists provided feedback that their preexisting workflows did not work well with the CMN because they needed to go to one place in the chart to find information to make a clinical decision regarding the patient's care, and then go back into the CMN to document their review. This feedback confirmed the need to develop reports that display directly in the CMN, so pharmacists would be able to easily evaluate patient data and document their review in one place. This revelation provided additional evidence of the utility of bidirectional feedback. The informatics team added the displays while clinical end-user training for the implementation was occurring, mitigating the chance of a negative user experience impacting uptake of the CMN, while concurrently avoiding delays in the project timeline. A full description of survey questions and responses is provided in [Table 3]. Recommendations provided were used to direct iterative build updates prior to enterprise-wide implementation.

Table 3 Pilot 2 survey themes

Survey question	Response themes
How do you think the Clinical Monitoring Navigator will help you?	Declutter the CML so new flags can be easily recognized Consolidation of pharmacist tasks into one location to improve the efficiency of patient reviews
How do you think the Clinical Monitoring Navigator will be detrimental to your work?	Since the pilot CMN does not display a comprehensive list of reports displaying patient data, pharmacists must go to one part of the chart to evaluate patient information and another to document their evaluation which is inefficient.
What, if anything, caused frustration when using the Clinical Monitoring Navigator?	The dynamic features of the CMN are useful to declutter the workspace and ensure that only things that need to be acted on are displayed, but it would be helpful to see rules that have previously been deferred in case the patient's clinical status changes.
Is there anything that was surprising or unexpected about the Clinical Monitoring Navigator?	It was a nice surprise to see that the rule sections were dynamic and only showed up when tasks needed to be completed.
What are your thoughts on the design and layout?	The design and layout flowed well It was easy to proceed through the navigator to complete tasks in a stepwise fashion
If you could change on thing about the Clinical Monitoring Navigator what would it be and why?	Add the ability to see previously deferred rules in case the follow-up plan changes Include additional reports and links so the patient review and documentation can all be completed in the CMN

Abbreviations: CML, Clinical Monitoring List; CMN, Clinical Monitoring Navigator.

Note: Pilot 2 focused on build changes to implement a Clinical Monitoring Navigator to provide dismiss and defer capability for preexisting Clinical Monitoring List rules. Navigator build was not comprehensive yet was intended to serve as a proof of concept to ensure that additional build aligned with pharmacist workflow needs. Responses were optimistic and suggested that the build of a comprehensive Clinical Monitoring Navigator would be well received and reflective of pharmacist monitoring needs.

Implementation

On March 5, 2019 the updated CML was implemented enterprise-wide as the primary patient monitoring tool for all inpatient pharmacists. Consistent with adequate end-user training and piloting using A/B testing, implementation was largely uneventful. Three help desk incident tickets were submitted to the project team for minor break/fixes. Written feedback from end-user pharmacists following the implementation was positive and consistent with the overwhelmingly positive feedback received during pilot testing.

For implementation, the prototype CMN was built out to include all 116 pre-existing CML rules with added additional display of relevant clinical information for each section ([Fig. 2]). The CMN was implemented for all inpatient pharmacists on April 14, 2020. Education was provided prior to implementation for a smooth transition for clinical pharmacists. The project team did not receive any tickets associated with the implementation of the CMN. Pharmacist feedback was positive and supported the streamlined view of top-priority clinical issues.

Discussion

Hospitalized patients require complex care by multidisciplinary teams. Pharmacists are tasked with monitoring patients' medication therapy to maximize benefit and minimize harm.[16] [17] [18] Adverse drug events significantly contribute to patient morbidity and mortality and represent a modifiable risk for patient harm.[19] [20] [21] CDS systems have the potential to enhance pharmaceutical care provided by pharmacists through assistance with identification of potential medication problems.[22] [23] Oftentimes pharmacists cover large numbers of patients. Therefore, the targeted rule logic to identify interventions for high-risk medications or patient-specific drug therapy problems can be critical for ensuring that pharmacists provide safe and effective care.[24] [25] [26]

As electronic health records become increasingly ubiquitous, an opportunity exists to optimize CDS functionality to assist clinicians in identifying and prioritizing work.[27] [28] [29] [30] [31] [32] [33] [34] When elegantly designed, scoring systems support clinicians in their day-to-day activities by presenting information in a deliberate, succinct, and actionable form.[27] Behind these systems, a paramount goal is to decrease burden on clinicians. Effective scoring systems are those that readily identify a standard set of potential problems, so clinicians can prioritize their cognitive effort on complex decisions that require individualized attention.[28] One method to achieve these objectives is through iterative, bidirectional end-user feedback. Our study demonstrated the utility of this through the findings of our end-user surveys. Upon initiation of the project, it was thought that the main outcome of the project would be a review and clean-up of the CML rules. Contrary to the initial plan, after obtaining pharmacist feedback, it became evident that pharmacists find little value in robust scoring system rule logic if there is no way to inactivate rules after they are evaluated and acted on. When assessed, the lack of the ability to interact with rules led to dissatisfaction and patient safety concerns because important flags could be easily overlooked. This dissatisfaction remained in place after the initial pilot involving revisions of the CML and was only remedied with implementation of rule dismissal and deferral associated with the CMN pilot. Intriguingly, despite this overwhelming feedback, the newly adopted electronic health record did not have “out of the box” dismissal and deferral functionality and therefore required significant amounts of custom build.

The success of this project was heavily influenced by the project management methodologies utilized. Our use of waterfall project management methodology facilitated stepwise development of the CML and CMN, allowing us to evaluate pharmacists' use and perception of each tool throughout the course of the project. The use of A/B testing allowed us to assess real-world use of proposed tools with a small group of pharmacists. This allowed us to quickly adjust build in accordance with their requests without the administrative and time burden of typical change request and education cycles, reducing the time to tangible change from months and weeks to hours or days. Lastly, the real-time feedback allowed us to use failure mode and effect analyses to identify risk points before expending large amounts of time completing build, ultimately reducing informaticist time spent performing build tasks and generating a more desirable product for end-users.

Our study is not without limitations. The intervention was limited to a single institution, yet the concepts and approaches remain applicable to similar projects at other institutions. Additionally, it was not feasible to perform direct observation of each pharmacist using the CML and CMN in daily practice. Direct observation would have provided the richest avenue for evaluating each pharmacist's feedback, but instead we needed to use surveys as a surrogate. Furthermore, our response rate for the initial enterprise-wide survey was 39%, indicating that there was a group of pharmacists that we were not able to attain feedback from. However, comparing this response rate to other pharmacist email surveys, we received many more responses than the typical 7%.[34] These findings may be suggestive that pharmacists were more willing to provide survey feedback because the functionality being evaluated has a significant impact on their day-to-day clinical activities. Lastly, because our EHR is unable to store information on rule firing, we are not able to quantitatively follow-up on rule outcomes, and have instead relied on qualitative end-user feedback to assess the utility of the rules on alert fatigue and drug therapy problem identification.

Feedback from this project provided evidence that iterative redesign of clinical monitoring and CDS tools is well received and can positively impact patient care activities. As we allow pharmacists to become more efficient, everyone benefits. Pharmacists serve as the drug therapy expert within the multidisciplinary team. Developing methods to improve end-user experience allows pharmacist to have the opportunity to investigate complex patient care decisions, provide increasingly comprehensive care, and take on additional patient care initiatives.

Conclusion

Clinical decision support represents a significant opportunity to enhance patient care, reduce adverse drug events and promote clinician efficacy. Stepwise evaluation of functionality gaps and provision of build adjustments allows for efficient progression toward project goals. This project demonstrates how close interaction between clinical end-users and project team informaticists can permit the successful implementation of CDS that aligns with patient care needs and clinical workflows.

Multiple Choice Questions

1. When elegantly designed, scoring systems support clinicians in their day-to-day activities by presenting information in ______________.

   • An alphabetical list

   • A deliberate, succinct, and actionable form

   • An interruptive, number coded, pop-up screen

   Correct Answer: The correct answer is option b.

2. What is one of the key reasons for iterative design planning for tools in the electronic health record?

   • To delay clinical decision support (CDS) as part of overall project implementation and allow developers more time to complete their build in the electronic health record.

   • To maximize the number of pop-up screens within the electronic health record that the clinicians are required to address for patient safety.

   • To allow bidirectional feedback between clinical end-users and project team informaticists to promote successful design and implementation of CDS.

   Correct Answer: The correct answer is option c.

Clinical Relevance Statement

The clinical relevance for this project is as an opportunity for other institutions to embark on similar evaluation of current workflows and electronic tools and design iteratively for overall improvements in the system. Effectiveness of electronic tools to support the practice as part of daily work leads to overall improvements in user satisfaction and minimizes clinician burnout related to system design.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

No human interventions were performed as the study iterations were based on the updates of the workflow and tools, rather than the direct patient care being provided.

• References

• 1 Hauck K, Zhao X. How dangerous is a day in hospital? A model of adverse events and length of stay for medical inpatients. Med Care 2011; 49 (12) 1068-1075
• 2 Robertson J, Walkom E, Pearson SA, Hains I, Williamsone M, Newby D. The impact of pharmacy computerised clinical decision support on prescribing, clinical and patient outcomes: a systematic review of the literature. Int J Pharm Pract 2010; 18 (02) 69-87
• 3 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
• 4 Garg AX, Adhikari NKJ, McDonald H. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 5 Yan L, Reese T, Nelson SD. A narrative review of clinical decision support for inpatient clinical pharmacists. Appl Clin Inform 2021; 12 (02) 199-20
• 6 Wilson JW, Oyen LJ, Ou NN. et al. Hospital rules-based system: the next generation of medical informatics for patient safety. Am J Health Syst Pharm 2005; 62 (05) 499-505
• 7 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 8 Shirley D. Project Management for Healthcare. 2nd ed.. Boca Raton, FL: CRC Press; 2020
• 9 Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017; 24 (01) 218-226
• 10 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 11 Golob Jr JF, Fadlalla AMA, Kan JA, Patel NP, Yowler CJ, Claridge JA. Validation of Surgical Intensive Care-Infection Registry: a medical informatics system for intensive care unit research, quality of care improvement, and daily patient care. J Am Coll Surg 2008; 207 (02) 164-173
• 12 Nelson SD, Del Fiol G, Hanseler H, Crouch BI, Cummins MR. A case report of refining user requirements for a health information exchange dashboard. Appl Clin Inform 2016; 7 (01) 22-32
• 13 Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care 1998; 7 (03) 149-158
• 14 Helfrich CD, Damschroder LJ, Hagedorn HJ. et al. A critical synthesis of literature on the promoting action on research implementation in health services (PARIHS) framework. Implement Sci 2010; 5 (01) 82
• 15 Kellum JA, Lameire N, Aspelin P. et al. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl 2012; 2 (01) 1-138
• 16 Kucukarslan SN, Corpus K, Mehta N. et al. Evaluation of a dedicated pharmacist staffing model in the medical intensive care unit. Hosp Pharm 2013; 48 (11) 922-930
• 17 Zhang C, Zhang L, Huang L, Luo R, Wen J. Clinical pharmacists on medical care of pediatric inpatients: a single-center randomized controlled trial. PLoS One 2012; 7 (01) e30856
• 18 Renaudin P, Boyer L, Esteve MA, Bertault-Peres P, Auquier P, Honore S. Do pharmacist-led medication reviews in hospitals help reduce hospital readmissions? A systematic review and meta-analysis. Br J Clin Pharmacol 2016; 82 (06) 1660-1673
• 19 Hakkarainen KM, Gyllensten H, Jönsson AK, Andersson Sundell K, Petzold M, Hägg S. Prevalence, nature and potential preventability of adverse drug events - a population-based medical record study of 4970 adults. Br J Clin Pharmacol 2014; 78 (01) 170-183
• 20 Wolfe D, Yazdi F, Kanji S. et al. Incidence, causes, and consequences of preventable adverse drug reactions occurring in inpatients: a systematic review of systematic reviews. PLoS One 2018; 13 (10) e0205426
• 21 Feng C, Le D, McCoy AB. Using electronic health records to identify adverse drug events in ambulatory care: a systematic review. Appl Clin Inform 2019; 10 (01) 123-128
• 22 Ibáñez-Garcia S, Rodriguez-Gonzalez C, Escudero-Vilaplana V. et al. Development and evaluation of a clinical decision support system to improve medication safety. Appl Clin Inform 2019; 10 (03) 513-520
• 23 Hincapie AL, Alamer A, Sears J, Warholak TL, Goins S, Weinstein SD. A quantitative and qualitative analysis of electronic prescribing incidents reported by community pharmacists. Appl Clin Inform 2019; 10 (03) 387-394
• 24 Roshanov PS, Fernandes N, Wilczynski JM. et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 2013; 346 (7899): f657
• 25 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 26 Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review. J Am Med Inform Assoc 2018; 25 (05) 593-602
• 27 O'Sullivan D, Fraccaro P, Carson E, Weller P. Decision time for clinical decision support systems. Clin Med (Lond) 2014; 14 (04) 338-341
• 28 Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform 2016; (Suppl. 01) S103-S116
• 29 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 26-30
• 30 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 31 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 32 Hsieh TC, Kuperman GJ, Jaggi T. et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004; 11 (06) 482-491
• 33 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 34 Hardigan PC, Popovici I, Carvajal MJ. Response rate, response time, and economic costs of survey research: a randomized trial of practicing pharmacists. Res Social Adm Pharm 2016; 12 (01) 141-148

Publication History

Received: 15 February 2021

Accepted: 17 May 2021

Article published online:
23 June 2021