Designing the User Interface of a Nitroglycerin Dose Titration Decision Support System: User-Centered Design Study

Abstract

Background Nurses adjust intravenous nitroglycerin infusions to provide acute relief for angina by manually increasing or decreasing the dosage. However, titration can pose challenges, as excessively high doses can lead to hypotension, and low doses may result in inadequate pain relief. Clinical decision support systems (CDSSs) that predict changes in blood pressure for nitroglycerin dose adjustments may assist nurses with titration.

Objective This study aimed to design a user interface for a CDSS for nitroglycerin dose titration (Nitroglycerin Dose Titration Decision Support System [nitro DSS]).

Methods A user-centered design (UCD) approach, consisting of an initial qualitative study with semistructured interviews to identify design specifications for prototype development, was used. This was followed by three iterative rounds of usability testing. Nurses with experience titrating nitroglycerin infusions in coronary care units participated.

Results A total of 20 nurses participated, including 7 during the qualitative study and 15 during usability testing (2 nurses participated in both phases). Analysis of the qualitative data revealed four themes for the interface design to be (1) clear and consistent, (2) vigilant, (3) interoperable, and (4) reliable. The major elements of the final prototype included a feature for viewing the predicted and actual blood pressure over time to determine the reliability of the predictions, a drop-down option to report patient side effects, a feature to report reasons for not accepting the prediction, and a visual alert indicating any systolic blood pressure predictions below 90 mm Hg. Nurses' ratings on the questionnaires indicated excellent usability and acceptability of the final nitro DSS prototype.

Conclusion This study successfully applied a UCD approach to collaborate with nurses in developing a user interface for the nitro DSS that supports the clinical decision-making of nurses titrating nitroglycerin.

Keywords

medication management - clinical decision support systems - graphical user interface - interfaces and usability - requirements analysis and design

Background and Significance

Nitroglycerin is a vasoactive medication for acute relief of chest pain.[1] There is no standardized dose of intravenous nitroglycerin because the medication's impact on blood pressure varies between patients.[2] While chest pain relief may be achieved for some patients at a low nitroglycerin dose, others require far greater doses to achieve the same outcome. Therefore, a trained nurse must manually increase or decrease the infusion (called titration) to achieve the desired dose.[2] [3] If the dose is not optimal, patients can experience side effects such as dizziness, light-headedness, nausea, vomiting, palpitations, headaches, and reflex tachycardia.[2] Adjusting the dosage is ultimately a trial-and-error process, and titrating incorrectly is common. In a descriptive study, only 8% of vasoactive titration events among 60 critical care patients were deemed titrated correctly.[4]

In our previous work, we developed a Nitroglycerin Dose Titration Decision Support System (nitro DSS) using data from the eICU database developed by Philips Healthcare. Specifically, we determined the accuracy of the machine learning algorithms used in nitro DSS to predict near-future blood pressure measurements for patients receiving nitroglycerin infusions.[5] A clinical decision support system (CDSS) helps in the decision-making related to patient care through targeted patient information, clinical knowledge, and relevant data.[6] CDSS applies data to algorithms and provides decision options based on the patient's specific circumstances.[7] The potential options are promptly delivered to clinicians. The nitro DSS predicts blood pressure to a potential new dose change of nitroglycerin, which could aid nurses' decision making for titrating nitroglycerin infusions. A nurse can input the prospective new dosage value into the system, and a predicted blood pressure would be generated. Based on this output and the nurse's clinical judgment, they can adjust the dose.[7] Consider a patient receiving a nitroglycerin infusion at 15 μg/min, who has ongoing chest pain rated at 10/10, and whose blood pressure is 130/80 mm Hg. The nurse considers increasing the dose to 20 μg/min to relieve the chest pain. The nurse would enter the potential new dose of 20 μg/min into the interface, and the predicted blood pressure would populate reading 128/78 mm Hg. The nurse can review the predicted blood pressure and apply their clinical judgment to decide whether to accept or reenter a different dose. CDSS developed for nursing tasks are designed to enhance nurses' decision-making.

The nitro DSS requires a user interface (visual display of the system) to enable interaction between nurses and the system. To design the interface, it is essential to focus on design features that make the system easy for the nurse to use.[8] Prior research has indicated that CDSS with critical design limitations, which were challenging to use, resulted in mistrust of the recommendations, leading to a low usage rate and increased user frustration.[9] These issues limit the potential utility of highly accurate predictive tools. For this reason, optimizing the design of the user interface is crucial for successful adoption.[10]

User-centered design (UCD) is a widely used approach that grounds the product design process on the needs and understanding of users.[11] UCD involves user engagement through interviews, surveys, and task analysis throughout the design process. Multiple iterative cycles of usability testing, which consists of a group of end users testing the prototype and providing feedback on the interface design, are usually performed to improve the interface design.[12] [13] [14] Several studies integrating a UCD approach in developing health technology products have shown interventions to be well-used and effective when tested.[15] [16] [17] [18]

Objective

The objective of this study is to design the nitro DSS user interface for titrating intravenous nitroglycerin doses through UCD principles.

Methods

To design the nitro DSS interface, a UCD approach consisting of two phases was conducted: (1) a qualitative study to identify design specifications for prototype development and (2) usability testing. In both phases, a demographic questionnaire was completed.

Qualitative Study to Identify Design Specifications for Prototype Development

The objective of the qualitative study phase was to understand the coronary care nurses' needs and preferences for the nitro DSS interface and translate those into initial design specifications. To achieve this objective, the interview session focused on two sections: (1) the cognitive task of titrating intravenous nitroglycerin infusions and (2) the interface design specifications. Cognitive task analysis involves uncovering and representing users' cognitive activities to perform specific tasks.[19] This technique was used to unpack the cognitive elements underlying nurses' decision-making when titrating nitroglycerin infusions. Understanding the nurses' cognitive process and decision-making can assist in designing the interface by ensuring all information and functions needed to titrate decision-making are included.[13] [19] Nurses were also asked to share their perceived wants and needs of nitro DSS while accounting for the identified constraint that it would be integrated into an existing system that contains blood pressure and nitroglycerin titration data (i.e., an electronic health record [EHR]), as this is a feasible implementation approach.

The semistructured interview guide was developed based on cognitive task analysis and literature about user interface design. Questions were extracted from an interview guide on exploring the decision-making of critical care nurses in performing titration of vasoactive drugs in cardiac patients for the cognitive task analysis.[20] The second section of the interview guide (interface design specifications) was developed to reflect the design and functional specifications for the decision support system.[21] All sessions of this study phase were virtual using Microsoft Teams at a time and date convenient for the participants. Interviews were audio-recorded.

The results of this phase identified initial design specifications for the nitro DSS interface, which were used to build the interface prototype using Figma, a tool for building interactive data-focused products.[22]

Usability Testing

Three in-person iterative usability testing cycles were conducted to test the prototype, and qualitative and quantitative data were collected. The results from each usability testing cycle are used to refine the interface and retest in the next cycle until prototype refinement is achieved (typically two to three cycles).[13] The UCD sessions were approximately 45 minutes each and held at the hospital at a convenient time and date for nurses. In each iterative usability testing cycle, the prototype was presented to the nurses on a laptop, and a brief overview of the functions of the developed nitro DSS was provided. Nurses then had the opportunity to interact with the prototype by entering potential new doses for which blood pressure predictions were generated.

In each round of usability testing, three titration examples were provided (see [Table 1]). Examples were intentionally created to gather insight into situations where nurses might consider predictions implausible given the current blood pressure and dose change. For instance, in round 2, nurses were instructed to enter a dose increase from 15 to 20 μg/min. The initial blood pressure was 134/77 mm Hg, and the predicted blood pressure with the dose increase was 175/88 mm Hg. In round 3, nurses were again presented with the blood pressure prediction of 175/88 mm Hg from the initial 90/60 mm Hg but for a dose change from 30 to 20 μg/min. Nurses were instructed to “think aloud” by expressing their thoughts, feelings, and opinions while interacting with the prototype. This approach is ideal for evaluating a prototype in the initial development stages because major usability problems can be identified with small samples.[19] Field notes were taken during the usability testing sessions.

Table 1 Actual and predicted blood pressure data for potential nitroglycerin dose (μg/min) of usability testing rounds

					Example 1	Example 2	Example 3
Testing round	Potential dose (μg/min)	5	10	15	20[a]	30[a]	20[a]
Round 1	Predicted blood pressure (mm Hg)	134/77	127/74	133/78	133/73	89/58	109/62
	Actual blood pressure (mm Hg)	144/80	128/78	134/77	133/73	90/60	110/60
Round 2	Predicted blood pressure (mm Hg)	134/77	127/74	133/78	175/88	89/58	109/62
	Actual blood pressure (mm Hg)	144/80	128/78	134/77	133/73	90/60	110/60
Round 3	Predicted blood pressure (mm Hg)	134/77	127/74	133/78	133/73	89/58	175/88
	Actual blood pressure (mm Hg)	144/80	128/78	134/77	133/73	90/60	110/60

^a Indicates the doses nurses entered into the Nitroglycerin Dose Titration Decision Support System prototype. Data from doses 5 to 15 μg/min was prepopulated on the interface.

Once the nurses tested the prototype, they were asked to complete quantitative measures consisting of two brief surveys: (1) The System Usability Scale (SUS) and (2) the Ottawa Acceptability of Decision Rules Instrument (OADRI). The SUS is widely used to evaluate the subjective usability of newly developed devices and systems.[23] [24] Usability is the extent to which a product can be used to achieve specified goals in a particular context.[25] The SUS is a simple, 10-item scale that yields valid and reliable scores among respondents in professional contexts. It has five response options ranging from strongly agree to strongly disagree.[24] This scale has garnered extensive use in assessing usability relating to health care applications.[26] [27] [28] OADRI is a validated instrument to measure the acceptability of clinical decision rules in the context of CDSS; therefore, it was selected for use in this study.[29] It is a 12-item instrument on a 6-point Likert scale (1 = strongly disagree, 6 = strongly agree). A neutral midpoint was included in the response options, allowing respondents to select no option/do not know. OADRI groups various barriers and facilitators related to whether a research innovation will be adopted into practice into three large categories: (1) aspects of the innovation: items 1 to 4, (2) decision-making: items 6 to 9 and 11, and (3) environment items 5, 10, and 12.[30] These categories are captured through questions such as: Is it easy to use, and is it useful in my practice? Do my colleagues support the use? Does the environment I work in make it difficult to use?[30] When it was developed, OADRI proved reliable (Cronbach's alpha >0.8). Assessing acceptability is a fundamental component of a UCD approach.[31] SUS and OADRI are brief questionnaires requiring less than 2 minutes each to complete.

Upon completing the questionnaires, a brief semistructured interview was conducted to elicit feedback on the nitro DSS interface and ways to improve the prototype (such as what nurses liked and disliked, how to improve, and how easy it was to use the interface). After completing the first cycle of usability testing, the results were used to refine the design and retest it in the second cycle. This process was repeated for a third cycle. Each cycle was completed when no new usability issues or changes were identified.[12]

Sample, Setting, and Ethical Approval

Convenience sampling was used to recruit nurses from adult coronary care units of two hospital sites from a single organization in southwestern Ontario, Canada. Participation in this study was extended to eligible registered nurses who met the following criteria: (1) currently employed in the coronary care unit and (2) had experience administering and managing intravenous nitroglycerin infusion and blood pressure monitoring. Nurses were permitted to participate in both phases of this study. However, efforts were taken to recruit a new group of nurses for usability testing. Ethical approval was obtained from the hospital's research ethics board and educational institution.

Data Analysis

A conventional qualitative content analysis approach was used for the interview data.[32] Analysis began after the first interview was completed. All data (summary notes and audio recordings from the session) were transcribed verbatim and checked for accuracy before analysis. Data were read word-for-word to derive codes.[33] The codes were organized into categories based on how different codes were related and linked. The categories were grouped into meaningful themes that identified the overarching design specifications of nitro DSS—NVivo (Version 12.7) assisted with the data analysis process.[34] Two researchers came together to confirm the codes and themes (investigator triangulation), ensuring multiple perspectives were incorporated during the analysis. Based on the results of the qualitative study, a digital interface prototype of the nitro DSS was developed.

Questionnaire scores from the SUS and OADRI were imported into Excel. Descriptive statistics were used to summarize scores from the SUS and OADRI. For the SUS questionnaire, the scores were converted to a range from 0 to 100, with 68 representing the minimally acceptable usability score.[24] The literature identifies a score of 68 as a useful benchmark (mean SUS score), where 50% of apps fall below and above this threshold.[24] [35] [36] [37] For the OADRI, items rated 1, 2, or 3 were rescaled to 0, 1, and 2, respectively. Responses “no opinion/do not know” were coded as the middle of the scale (score 3), as done in previous research.[29] [38] The responses were reversed for items 8 to 12, so higher scores always represent higher acceptability. The overall acceptability was calculated by averaging the score of all 12 items, resulting in a score that ranged from 0 (lowest) to 6 (highest) acceptability, with an overall mean score >3 indicating good acceptability.[29] A plot of the mean SUS and OARDI scores over the three rounds was created to compare the usability testing cycles visually.

Results

Study Participants

A total of 20 nurses participated in this study, with 7 nurses for the qualitative study phase and 15 for the usability testing phase (5 in each of the three cycles). Two nurses participated in both phases of this study. Most of the nurses were female (86%), ranging from 25 to 34 years of age (46%). The average experience as a coronary care nurse was approximately 10 years (ranging from 1 to 26 years), with most nurses having a bachelor's degree (86%) and the remaining having a diploma (14%). The average experience with CDSSs was 2.8 years (range 0.6–10 years), and they rated themselves as competent (41%), proficient (36%), or expert (18%) in using CDSSs. A summary of participant characteristics is presented in [Table 2].

Table 2 Participant demographics and characteristics

Characteristics	Total participants (n = 22)	Phase 1 participants (n = 7)	Phase 2 participants (n = 15)
Age (y), n (%)
25–34	10 (46%)	2 (28%)	8 (53%)
35–44	8 (36%)	4 (57%)	4 (27%)
45–54	2 (9%)	1 (14%)	1 (7%)
55–64	2 (9%)	0	2 (13
Female, n (%)	19 (86%)	7 (100%)	12 (80%)
Highest level of education completed, n (%)
Diploma	3 (14%)	2 (29%)	1 (7%)
Bachelor	19 (86%)	5 (71%)	14 (93%)
Work experience (y), mean (range)	9.8 (1–26)	11 (1–26)	9.2 (2–23)
Frequency of nitroglycerin administration/management, n (%)
Never	0	0	0
Rarely	11 (50%)	4 (57%)	7 (47%)
Sometimes	9 (41%)	3 (43%)	6 (40%)
Often	2 (9%)	0	2 (13%)
Always	0	0	0
Proficiency in managing nitroglycerin infusion, n (%)
Novice	0	0	0
Advanced beginner	3 (14%)	0	3 (20%)
Competent	12 (55%)	4 (57%)	8 (53%)
Proficient	6 (27%)	3 (43%)	3 (20%)
Expert	1 (4%)	0	1 (7%)
Experience with CDSS (y), mean (range)	2.8 (0.58–10)	2.5 (2.5–2.5)	3.2 (2.5–10)
Proficiency using CDSS, n (%)
Novice	0	0	0
Advanced beginner	1 (5%)	1 (14%)	0
Competent	9 (41%)	1 (14%)	8 (53%)
Proficient	8 (36%)	5 (72%)	3 (20%)
Expert	4 (18%)	0	4 (27%)

Abbreviation: CDSS, clinical decision support system.

Identifying Design Specifications for Prototype Development

Four themes captured the design specifications for the nitro DSS. These included (1) Clear and Consistent Interface, (2) Vigilant, (3) Interoperable, and (4) Reliability. Exemplar quotes supporting the themes are provided in [Table 3].

Table 3 Exemplar quotes related to the identified themes from the qualitative study phase

Theme	Quotes
Clear and consistent interface	I think to keep it as simple as possible—white background. I'm thinking keep it the same or very similar fonts and appearance to the current EHR. So, I think the less different this new system is from (current EHR), the better, because if it's very different, we have to take more time to find the information we need. But if it's designed to be similar and something that we're used to, we know, like your eyes and your brain processes it easier and faster. (Participant 2) I would want it to be similar to (current EHR) because it autopopulates under the medication administration record in (current EHR). (Participant 6) I want it to be user-friendly. I want the screen to be as simplified as possible. I don't want too much stuff on it. I want it to be easy to read. I'm keeping in account that we can see the information easily. It's not very difficult to find or overwhelming. (Participant 6) Some patients just aren't quite that straightforward. Sometimes it is good to know the trend from when they first came in. So, if there was a way to keep track of that and then have it easily accessible, I don't know if it needs to be on that screen, but accessible might be helpful. It would be best if you were straightforward, as that's the important thing, not too many words or too many visuals. I find, especially in nursing, you're in a hurry to have the information you need without many extras. (Participant 7) For the blood pressure trend, it could be something if you hover on the past value that's already there, that it pops up, and you see the trend graph of how it went. Otherwise, keep it very simple. (Participant 4)
Vigilant	The Nitroglycerin order is going to come with some parameters. I want an alert when the predicted blood pressure falls below those parameters. (Participant 2) You don't want people to ignore it (alert) and you don't want to have it a pop-up like other pop-ups in (current EHR) where you can't continue charting if you don't address it. (Participant 3) If an alert is out of range, alerts are valuable. I think it just needs to be not too many. It needs to alert us when we need to be aware, but there are so many that we end up blocking it out. So, if it's a blood pressure we must address, I think having an alert makes sense. (Participant 7)
Interoperable	It would be best (to integrate) in the MAR. Logically, uptake is best in the MAR because it will all be done in one place. (Participant 6) I would want this system to give a predictive blood pressure for a potential dose, and you click OK, then it flips it into your documentation. I don't want to have to hit OK and then document. (Participant 1)
Reliability	I want to see how accurate that's (prediction) has been. If it's giving me an answer, I can trust it because what it told me in the past has been accurate. (Participant 3) It would be good to know (the performance of the system), and it's nice to have confidence in the system you're using to know how accurate it is. If I found that whatever titration I'm doing up or down is not accurate, and I'm making decisions on something that I don't have confidence in that's going to be accurate or at least somewhat accurately predict, then I would start to lose confidence (in the system). But then the opposite would also be true. If I find that the more I use it, the more I find that it's pretty accurate in predicting, the more confidence I would have in using it. (Participant 7) It would be too much on the screen, but I don't agree with just blindly trusting. Could that also be like a hyperlink? If you have suspicions or are wondering about it, you can look into it, but it won't necessarily interrupt your workflow. (Participant 5) A graph would be helpful. To see them both drawn out. Like if I could have the option to have a graph because, at a glance, it would be very helpful over 12 hours not to go through, you know, every 15 minutes of blood pressure predicted or actual. I would like to see a graph, which would give me a much quicker snapshot of its accuracy (the prediction). (Participant 2)

Abbreviation: EHR, electronic health record; MAR, medication administration record.

Clear and Consistent Interface

A clear and consistent interface refers to the design and presentation of the interface in a way that is easily understandable and uniform with existing interfaces, creating an intuitive experience. Nurses in this study frequently referred to their current EHR system when asked how the nitro DSS user interface should be designed. They requested that the nitro DSS interface be consistent with their EHR; for example, they asked for the physiological data to be presented in a flowsheet, as currently shown in the EHR. Consistency with the EHR makes the nitro DSS interface more straightforward and requires less cognitive load. This was translated to a design requirement by presenting vital signs, chest pain score, side effects, and nitroglycerin dosage data in a flowsheet format. Statements from nurses that highlighted the preference for an interface consistent with the EHR are presented in [Table 3].

The theme of a clear and consistent interface also alludes to the requirement for the interface to avoid information overload. Nurses emphasized this preference through repeated descriptions of the burden and adverse impacts of visual overload. They asked for the user interface of the nitro DSS to present only the information necessary for titration-related decision-making, and [Supplementary Material] (available in the online version) can be included as hyperlinks or in an “additional resources” section. Nurses described this as the difference between the “I need to know” and “good-to-know” information. For example, nurses shared the “need-to-know” information, including vital signs, dose-related data, and chest pain scores, as this information is essential to deciding when to increase or decrease the dosage. Nurses shared that “good-to-know” information, like the nitroglycerin protocol itself, should be included as a link rather than described on the main page. When asked how they would want the content to be presented to showcase a clear interface, the nurses said they wanted to keep it simple and display essential information in a table format. They declined the need for all vital signs or chest pain scores to be presented in a graph. Instead, they explained that while the blood pressure graph is valuable information, it should not be on the main screen and only be accessed when necessary.

Vigilant

A vigilant interface actively monitors patient predictions, prioritizes alerts based on deviations from normal ranges, and aims to enhance patient safety without burdening nurses with excessive or unnecessary alerts, thereby addressing the issue of alert fatigue in health care. Nurses expressed a need for nitro DSS to alert nurses of abnormal predictions that pose a risk to patient safety, but at the same time, they were wary of the potential impact of excessive alerts on alert fatigue (see [Table 3]). The nurses claimed they were already overloaded with alerts in their practice. As such, there was a need for nitro DSS to have a vigilant interface by balancing alert fatigue and patient safety, meaning a visual alert appears on the screen only when the expected blood pressure deviates from the desired range, which is defined in nitroglycerin protocol as systolic blood pressure less than or equal to 90 mm Hg.[39] To reduce fatigue, no auditory alarm was developed.

Interoperable

An interoperable system can exchange data seamlessly and efficiently and interact with other systems to work together and share information to enable efficiency. Nurses suggested that the system be integrated into the medication administration record (MAR) component of the EHR, as this is where nitroglycerin dose changes are documented. This would allow information on dosing to be automatically entered in the MAR, reducing double documentation (see [Table 3]). Specifically, they suggested that once a dose is accepted on the nitro DSS interface, this should automatically translate into the MAR, as it will save time and improve efficiency.

Reliability

Reliability is defined as consistently generating accurate predictions, demonstrating high trustworthiness in its predictive capabilities, and having transparency in its performance. Nurses expressed a need to know if the system was reliable. Precisely, they needed to understand the accuracy of the blood pressure predictions. Knowing the system was reliable would help nurses develop trust and potentially increase the use of the nitro DSS. When asked how nurses would like the accuracy of the predicted blood pressure to be presented, nurses shared that they would like a graph showing the actual and predicted blood pressure over time (see [Table 3]).

Usability Testing

[Table 4] summarizes the interface development. [Figs. 1] and [2] depict the SUS and OADRI scores collected from the participants throughout the three usability rounds. [Fig. 3] shows an overview of the prototype versions developed based on the usability testing results.

Fig. 1 Results of the System Usability Score. Gray bars are individual participant scores. Orange bars are the mean score for each round of usability testing. Blue bars are the overall score across all three rounds. A score >68 indicates the above-average usability of a system. SUS, System Usability Score.

Fig. 2 Results of the Ottawa Acceptability of Decision Rules Instrument score. Gray bars are individual participant scores. Orange bars are the mean score for each round of usability testing. Blue bars are the overall score across all three rounds. OADRI, Ottawa Acceptability of Decision Rules Instrument.

Fig. 3 Overview of the nitro DSS prototypes. Red arrows indicate the changes to the nitroglycerin protocol feature. The red numeric values in figures from rounds 2 and 3 and the final version represent abnormal blood pressure measurements. Changes to the side effects row can be seen in prototypes tested in rounds 3 and 4. nitro DSS, Nitroglycerin Dose Titration Decision Support System.

Table 4 Summary of the iterative development process

Round	Express user need	Design modifications
Initial prototype	Based on the results of Phase 1  • Clearer cues on where to enter the numeric input for new dose  • Clearer cues on where to enter the data entry features placement  • Consistent placement of “Accept dose” or “Cancel dose” options to match the hospital EMR  • Have the blood pressure graph as an optional feature so users can quickly identify patient trends  • Need for data on vital signs, chest pain score, and nitroglycerin dosage presented in a table format with time navigation  • Clear icon use for nitroglycerin protocol  • Nee for an alert when low systolic blood pressure prediction is below the threshold of 90 mm Hg	• Be able to enter a numeric value in the potentially new dose rather than preselected options. The predict button should be green. The predicted blood pressure should be in blue font color  • Having all data entry features at the top  • The accept dose or cancel features in the bottom right corner are blue  • Having a blood pressure graph displaying actual versus predicted blood pressure over time as a feature. But not displayed on the main screen. This would be an option the user can select to view if needed. This feature can be called Actual  • Having a table that reflects the current flowsheet in the current EHR displaying the vitals (blood pressure, heart rate, Resp., SpO2), chest pain score, and nitroglycerin dose. The table only shows five timeslots at a time but can have a slider feature where they can slide to see the different data in the time columns, that is, I can move to see information at 1,135 but move back to see 1,100 information.  • Having a “paperclip” icon showing an attachment to indicate nitroglycerin protocol • Generate an alert for blood pressure predictions of systolic below 90 mm Hg. This prediction would appear below the predicted blood pressure stating, “The system predicted a critically low blood pressure.” Have the option to select okay/close the alert
Round 1 (version 2)	• Inclusion of mean arterial pressure in predicted blood pressure  • Removal of irrelevant cues from the flowsheet  • Increased clarity of nitroglycerin protocol feature  • Clearer identification of abnormal values  • Enhanced blood pressure graph readability  • Clarity in feature placement	• Add mean arterial pressure beside blood pressure in table  • Delete RR from the flowsheet  • Change the attachment icon to Nitroglycerin Protocol  • Add a nitroglycerin dose graph  • Make abnormal values in red (the blood pressure value of 90/60 in chart in red as “!90/60” and heart rate to !110 from 110)  • In the graph add numeric values for blood pressure and time, that is, @1100: 144/80  • Make the abnormal blood pressure alert appear in the center of the screen rather than right under the blood pressure. The rest of the screen should be greyed out and only the alert is displayed.  • Move blood pressure and predicted blood pressure closer to the center rather than on the right side.  • Once you press accept dose that option disappears  **Incorporated a scenario to have the dose change from 15μg/min with a blood pressure of 134/77 mm Hg to 20 μg/min with a predicted blood pressure of 175/88 mm Hg for the first prediction
Round 2 (version 3)	• Relocate “nitroglycerin protocol” icon and include descriptive label  • Increased clarity in displaying abnormal values  • Increased clarity in identifying graph icons  • Add “side effects” row in chart  • Display nitro DSS performance information in blood pressure graph window  • Enhance graph readability by displaying numeric values  • Include map with predicted blood pressure	• Move the “nitroglycerin protocol” icon to the top right corner and change from Nitroglycerin Protocol to Link to Nitroglycerin  • Remove ! for abnormal values  • Instead of word “Actual” use graph icon. Also position this icon in the same area as blood pressure (MAP)  • Create another row in the chart that is “side effects” In this, it is a free text option for nurses to add info  • In the blood pressure graph window included the nitro DSS' performance on patients from that hospital. Included, “Overall absolute error is 5 mm Hg” and an information icon that when clicked it states, “This is the error rate (difference between actual and predicted blood pressure) for the system when tested on 1,000 THP patients.”  • In the blood pressure graph, include the numeric value for actual and predicted blood pressure when hovered over the different time intervals.  • Include mean arterial pressure into the prediction so it appears with the blood pressure  **Incorporated a scenario to have the dose change from 15 μg/min with a blood pressure of 134/77 mm Hg to 20 μg/min with a predicted blood pressure of 133/73 mm Hg for the first prediction. For the third prediction the blood pressure was 90/60 mm Hg with a dose of 30 μg/min and increased to 175/88 with a dose change to 20 μg/min
Round 3 (version 4)	• Clarify nitroglycerin protocol label  • Streamline side effects options  • Include nitroglycerin start date/time  • Change blood pressure graph icon color  • Option for predicted blood pressure display  • Revise “cancel dose” to “reject” with free text	• Make the Link to Nitroglycerin Protocol appear like a hyperlink (blue + underline)  • Make the side effects a drop-down option  • Diaphoresis  • Dizziness  • Headache  • Vision changes  • Other_____________  • Include Nitroglycerin Start Date/Time: August 6, 2023, @1100 under the medication name  • Change blood pressure graph icon to color blue  • Remove “X” on the alert pop-up as it has same purpose as okay option  • Created an option to show predicted blood pressure below the actual blood pressure in the table. Users can select if they want to view predicted blood pressure and unselect if they do not.  • Changed the cancel dose option to reject and nurses can free text why they are rejecting the potential new dose/predicted blood pressure

Abbreviation: EHR, electronic health record; nitro DSS, Nitroglycerin Dose Titration Decision Support System; RR, respirarion rate.

Usability

All the participants' ratings on SUS exceeded the threshold score of 68. Overall, the usability score decreased slightly after the second round of usability testing (from mean of 90.5 to 86). Additional features were incorporated and tested in round 2 based on the feedback from round 1. However, some nurses shared that these changes made the prototype more complex. For instance, nurses found the Nitroglycerin Protocol icon ambiguous and suggested it be better identified and separated from the accept and cancel buttons (see [Fig. 3]). This feature was revised to include a descriptive label, Link to Nitroglycerin Protocol, placed at the top right corner, and the reaction from the nurses was favorable to this modification (see [Fig. 3]). The nurses also reported the need to incorporate a function to record patients' side effects. Initially, this was designed as a free-text feature. However, nurses desired a drop-down menu to select side effects rather than manually entering the side effects (see [Fig. 3]). These findings in the usability testing aligned with the theme of a clear and consistent interface as identified in the qualitative study phase.

Another feature from round 2 that nurses wanted modified to improve usability was the presentation of abnormal values. The prototype in round 2 identified all predictions of blood pressure that fell outside normal limits in red color and with an exclamation mark (i.e., !90/60). The nurses found this change misleading as the exclamation mark can be mistaken for the number 1 reading, a blood pressure of 190/60. Abnormal values were revised to remove the exclamation mark (i.e., 90/60; see [Fig. 3]). Furthermore, in round 2, the word Actual was used to indicate the presence of a graph that would be accessible to nurses when they clicked on this feature. However, using the word, Actual, confused the nurses as this term does not inherently symbolize or imply the presence of a graph in a typical user interface. Nurses suggested changing Actual to a graph icon for clarity (see [Fig. 3]).

Another potential contributing factor to the reduced usability score in round 2 is that a blood pressure prediction that the nurses may consider unlikely, given the current blood pressure and potential change in dose, was introduced into the prototype. The blood pressure was 134/77 mm Hg at 15 μg/min, and with a dose increase to 20 μg/min, the predicted blood pressure was 175/88 mm Hg. The requirement to consider situations where the decision support system provides a prediction that does not align with their intuition yielded some valuable insights that enhanced the user interface. Nurses suggested including the predicted mean arterial pressure to help assess whether to accept or reject the dose. The nurses also requested a feature that identifies the mean absolute error (difference between the actual and predicted blood pressure) related to nitro DSS's performance within their institution to determine the practical use of the prototype (see [Fig. 4]). These insights during the usability testing phase reflected the theme of reliability. Overall, the inclusion of this scenario in round 2 prompted nurses to scrutinize the utility of nitro DSS, which, in turn, fostered valuable input aligned with the theme of reliability, leading to improved usability of nitro DSS.

Fig. 4 Displays the original and final version of the graphs for the nitro DSS prototypes. The original version (on the left) includes a graph of the actual and predicted systolic blood pressure over time. The final version (on the right) includes the actual and predicted systolic blood pressure, a graph of the nitroglycerin dose over time, and the absolute error rate for nitro DSS when tested on patients from that organization. BP, blood pressure; nitro DSS, Nitroglycerin Dose Titration Decision Support System.

Acceptability

The acceptability scores for each participant and the mean score from each round of usability testing are reported in [Fig. 2]. The scores of the OADRI showed a trend of increasing average acceptability, consistent with participant responses during interactions with consecutive prototypes and comments during the interview, indicating reduced barriers and enhanced facilitators related to whether nitro DSS would be adopted.

All the nurses confirmed that they had not previously encountered a CDSS for titrating nitroglycerin doses. It was their first exposure to a system that predicted a physiological indicator for a titration task. Based on the items in the OARDI, all nurses moderately or strongly agreed that patients would benefit from using the decision support system, the system's wording is unambiguous, and it would be useful for their practice.

For nurses, a key determinant of accepting nitro DSS was ensuring a low error rate between the actual and predicted values. During the debrief interview, nurses said there is little value in adopting nitro DSS in their practice if it is inaccurate. Nurses wanted evidence about the accuracy of the predictions within their institution and patient population rather than academic papers. Nurses conveyed that knowing nitro DSS's performance within their institution is crucial, as it assists in making an informed decision about using it in their clinical practice.

Nurses also expressed the importance of a feedback mechanism to reduce further the error rate between actual and predicted blood pressure in order to accept the CDSS. A feedback mechanism for continual improvement of the prediction component allows users to comment on the prediction, making it easier to gather user input to improve the performance of nitro DSS. A new feature that authorized nurses to reject the prediction, accept the dose, and share their reason for rejecting it was added (see [Fig. 5]). Nurses wanted this feature to help train the algorithm and improve the overall performance of nitro DSS. This design modification reflected the theme of reliability, and nurses appreciated having this feature to support the acceptability of nitro DSS.

Fig. 5 Displays the feature to report the reason for rejecting the prediction (final prototype; data are imaginary).

Nurses expressed having minimal data entry requirements that would encourage them to adopt nitro DSS. We translated this need by including three data entry features: potential new dose, patient side effects, and reason for rejection. As mentioned previously, patient side effects were revised to have a select all that apply feature, and reason for rejection is not required for each output.

Having aesthetically appealing alerts to catch attention was mentioned as a determinant of acceptability during the debriefing interview. In usability testing round 1, the nurses shared that they disliked the critically low blood pressure alert on the right side of the screen, as it was not eye-catching. The prototype was modified to place the critically abnormal blood pressure alert in the center of the screen (see [Fig. 6]). Alluding to the theme of a vigilant interface from the interviews, this change was sustained in all subsequent rounds of usability testing, and nurses shared positive feedback and appreciated the revision.

Fig. 6 Displays the alert for abnormal blood pressure prediction on the original (top) and final (bottom) prototypes (data are imaginary).

Discussion

Our study findings highlighted interoperability as an essential aspect of designing a CDSS. Nurses expressed the need for nitro DSS to communicate with clinical information systems, specifically EHR, and exchange information seamlessly. Consistent with prior research, interoperability features are integral to developing and implementing CDSSs.[40] [41] A qualitative study explored the challenges in developing, managing, and using CDSSs, and interoperability was identified as a requirement by the nurses to reduce the burden of manual data entry and prevent double documentation.[42] Minimizing data entry requirements decreases the time spent using the system, allocating more time to direct patient care.[43] Furthermore, clinicians and CDSS vendors are frustrated with the minimal progress made to have CDSS interoperable with EHRs[42]: These findings add to the present study the importance of collaboration between CDSS developers and EHR vendors to ensure interoperability. Double documentation burden could be reduced if nitro DSS and existing systems exchange information; for example, patients' vital signs can be imported from the EHR, and dosages can be translated into the MAR, rather than nurses manually inputting this information. Reducing double documentation minimizes repetitive tasks that are time-consuming and prone to error and can lead to inconsistencies or discrepancies in data, ultimately creating inefficiencies.[43]

Ensuring interoperability among systems could enhance the accuracy of nursing documentation. Tung et al (2022) evaluated timestamp discrepancies between EHR documentation and pump event logs (actual administration process) for vancomycin infusion.[44] The researchers used process mining to examine the conformance between pump event logs and EHR data for a single hospital in the United States.[44] An algorithm was developed to link records of the same infusions, and discrepancies in infusion start time, completion time, and interruptions were analyzed. Of the 1,858 infusions, approximately 20% had an infusion start time discrepancy of more than ±10 minutes. The researchers suggested these discrepancies are inherent parts of the current EHR documentation process commonly found in hospitals, not unique to the study site.[44] A potential solution to reducing the discrepancy in documentation can be to have interoperable systems. Suppose a dose accepted on nitro DSS is automatically documented in the MAR section of the EHR; this could eliminate the need to document it in the EHR and potentially reduce the time discrepancies.

Nurses possess a wealth of knowledge and expertise that can be leveraged to create more tailored and effective CDSS.[45] [46] Since nurses will be the end-users of nitro DSS, their involvement in the development process could significantly shape the acceptance and use of nitro DSS in clinical practice. In our study, nurses identified essential design specifications of nitro DSS. Nurses shared that if the interface is centered on their needs, it will be easier to use and more prone to adoption in their clinical practice. Incorporating user-identified preferences into user interface design aligns with the insights from the technology acceptance model (TAM), a well-recognized conceptual model used for information systems research.[47] TAM is used to understand why users accept or reject information technology.[48] TAM suggests that individuals' beliefs about the usefulness and ease of using technology have led users to (1) form positive attitudes toward technology, (2) develop intentions to use that technology, and (3) actually use the technology.[47] [48] Not having the end users involved in artificial intelligence technology development can lead to interfaces that inadequately capture users' requirements, potentially resulting in low adoption rates.[49] SUS is often used to assess a system's perceived ease of use, and this study's results indicate excellent usability of nitro DSS.[50] According to the TAM, perceived ease of use is a precursor to technology acceptance.[51] Therefore, the results denote a favorable foundation for the technology's eventual use.

Nurses' involvement in usability testing research can provide an opportunity to understand the prototype's functionality and purpose, potentially reducing emotions of fear, doubt, and frustration tied to using decision support systems in practice.[52] Previous work highlighted nurses' resistance to adopting CDSSs in their practice, demonstrating that they would repeatedly double-check the reliability and credibility of the CDSS because they doubted the algorithm's accuracy.[52] In our study, the reliability of the CDSS was also a theme identified from the qualitative study findings and an essential component of usability and acceptability. Their involvement allowed them to share preferences of design specifications that can be incorporated into nitro DSS to develop trust. A critical factor in adoption is trust, which can be bolstered through transparent information on underlying evidence, collaboration with end-users, and feedback about users to drive continuous improvement.[53] Nurses wanted to know that blood pressure predictions within their organizations were accurate, as this can help them to make an informed decision on whether to accept or reject those predictions. Clinicians' trust in predictive CDSSs is influenced by their perceptions of the accuracy and correctness of the CDSSs, and the more accurate the prediction, the greater the trust in the CDSS.[55] For the nitro DSS, a graph showing the actual and predicted blood pressure over time and mean absolute error (difference between the actual and predicted blood pressure) related to the nitro DSS model performance was embedded into the interface. Furthermore, the involvement of clinicians in developing the CDSS is an essential factor contributing to trust. Clinicians' ability to understand the system (perceived understandability) is also an essential factor influencing trust.[54] CDSS developers need to involve end-users in developing CDSS prototypes and integrating features that demonstrate the system's reliability to gain clinicians' trust.

Limitations

The study sample included a relatively small number of nurses from two separate hospitals of a single organization in Canada. Nurses with various years of work experience helped design the user interface, but only nurses who worked in the coronary care department were included. Nurses in other departments, such as the emergency department, use nitroglycerin for angina relief, so their preferences for nitro DSS may differ. Furthermore, nurses in this study highlighted the importance of aligning the nitro DSS interface with their existing EHR, recognizing other organizations may not use the same EHR system. Readers should reflect on these factors when considering the transferability of the findings to other contexts.

The interviewer was a research team member who designed the initial prototype based on results of phase one. It is possible that nurses were hesitant to share opinions criticizing the system. The interviewer reassured the nurses that they could openly share their feedback on the system as it would help to improve the design.

In this study, three rounds of usability testing were conducted to develop a nitro DSS prototype. The subsequent research should include developing a high-fidelity prototype, testing in silent trials, and conducting a randomized control trial. Additional research with a larger, more diverse sample of nurses from multiple organizations across Canada, employed in different clinical settings, and using various EHRs is needed to develop a high-fidelity nitro DSS prototype and test usability and acceptability further. Once a high-fidelity prototype is developed, the accuracy of blood pressure predictions generated by nitro DSS in real-time can be evaluated using silent trials. The benefit of silent trials is that models are evaluated on prospective patients in real time while the end-users are blinded to the model output (blood pressure predictions), meaning their clinical decision-making is unaffected.[55] The next step would be to conduct a randomized controlled trial in the coronary care units to assess the accuracy and use of nitro DSS. The measures evaluated would include the number of hypotension and hypertensive episodes, the number of titrations required to achieve relief of angina, and nurses' nitro DSS usage patterns (i.e., number of missed alerts or delays as measured in minutes in responding to alerts, usability, and acceptability).

Conclusion

Through a UCD approach, this study developed a user interface prototype that interacts with nurses to titrate intravenous nitroglycerin dosage. This study used an in-depth qualitative phase to pinpoint design specifications to inform a prototype that later underwent three iterative usability testing rounds to enhance the user interface. The refined nitro DSS prototype incorporates features like side effect reporting, easy access to a summary measure of the blood pressure prediction accuracy for the local context, a reporting mechanism to record if a user considered the blood pressure prediction was unlikely to be correct, and minimal alerts to prevent the burden of alert fatigue. The study highlighted that nurses found the final nitro DSS prototype to have excellent usability and acceptability. Further research is required to evaluate the usability and acceptability of nitro DSS in direct patient care. The involvement of nurses in user interface development is crucial, as demonstrated by this study, which showed that their input led to a usable and acceptable prototype.

Clinical Relevance Statement

Nurses' involvement in interface development is essential to generate design specifications that enhance the usability and acceptability of CDSSs. The final iteration of the nitro DSS prototype incorporates nurses' needs for a reliable, interoperable, vigilant system with a clear and consistent interface. This study can guide designing the interface of CDSSs for titrating other vasoactive agents.

Multiple-Choice Questions

1. What is a think-aloud technique?

   • Users express their thoughts, feelings, and opinions while interacting with the prototype.

   • Observing individuals performing their typical clinical workflows.

   • Patient charts are gathered to answer a clinical question.

   • Participants complete a survey.

   Correct Answer: The correct answer is a. The think-aloud technique involves participants talking aloud while interacting with the prototype and expressing their user interface perspective.

2. What is the definition of a clear and consistent interface as defined in this study?

   • Actively monitors patient predictions, prioritizes alerts based on deviations from normal ranges, and aims to enhance patient safety without burdening nurses with excessive or unnecessary alerts, thereby addressing the issue of alert fatigue in health care.

   • Design and present the interface in an easily understandable and uniform way with existing interfaces, creating an intuitive experience.

   • Exchange data seamlessly and efficiently, interact with other systems to work together, and share information to enable efficiency.

   • Consistently generating accurate predictions, demonstrating a high level of trustworthiness in its predictive capabilities, and having transparency in its performance

   Correct Answer: The correct answer is b. A clear and consistent interface refers to the interface's design and presentation of the interface, and it is designed in a way that makes it easily understandable and uniform with existing interfaces, thereby creating an overall intuitive experience.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the hospital and educational institute of the corresponding author.

• References

• 1 Tarkin JM, Kaski JC. Pharmacological treatment of chronic stable angina pectoris. Clin Med (Lond) 2013; 13 (01) 63-70
• 2 Cheng DCH, Martin J, David T. Evidence-Based Practice in Perioperative Cardiac Anesthesia and Surgery. Springer; 2021
• 3 Fadale KL, Tucker D, Dungan J, Sabol V. Improving nurses' vasopressor titration skills and self-efficacy via simulation-based learning. Clin Simul Nurs 2014; 10 (06) e291-e299
• 4 Arellano D. Vasopressor Titration in Medical Intensive Care Patients with Septic Shock: A Descriptive Study. 2019
• 5 Kamboj N, Chang K, Metcalfe K, Chu CH, Conway A. Accuracy and precision of continuous non-invasive arterial pressure monitoring in critical care: A systematic review and meta-analysis. Intensive Crit Care Nurs 2021; 67: 103091
• 6 Osheroff JA, Teich JM, Levick D. et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. CRC Press; 2012
• 7 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3 (01) 17
• 8 Vaghefi I, Tulu B. The continued use of mobile health apps: insights from a longitudinal study. JMIR Mhealth Uhealth 2019; 7 (08) e12983
• 9 Goh WP, Tao X, Zhang J, Yong J. Feature-based learning in drug prescription system for medical clinics. Neural Process Lett 2020; 52 (03) 1703-1721
• 10 Tang T, Lim ME, Mansfield E, McLachlan A, Quan SD. Clinician user involvement in the real world: designing an electronic tool to improve interprofessional communication and collaboration in a hospital setting. Int J Med Inform 2018; 110: 90-97
• 11 Nguyen Ngoc H, Lasa G, Iriarte I. Human-centred design in industry 4.0: case study review and opportunities for future research. J Intell Manuf 2022; 33 (01) 35-76
• 12 Barnum CM. Usability Testing Essentials: Ready, Set…Test! Morgan Kaufmann;. 2020
• 13 Riihiaho S. Usability testing. Wiley Handb Hum Comput Interact. 2018; 1: 255-275
• 14 Witteman HO, Vaisson G, Provencher T. et al. An 11-item measure of user-and human-centered design for personal health tools (UCD-11): development and validation. J Med Internet Res 2021; 23 (03) e15032
• 15 Tsimicalis A, Stinson J, Thorstad K. et al. The design, development, and usability testing of an ehealth program for youths with osteogenesis imperfecta: protocol for a 2-Phase User-Centered Mixed Methods Study. JMIR Res Protoc 2023; 12 (01) e47524
• 16 Chu CH, Biss RK, Cooper L, Quan AML, Matulis H. Exergaming platform for older adults residing in long-term care homes: user-centered design, development, and usability study. JMIR Serious Games 2021; 9 (01) e22370
• 17 Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Mobile phone-based telemonitoring for heart failure management: a randomized controlled trial. J Med Internet Res 2012; 14 (01) e31
• 18 Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res 2012; 14 (03) e70
• 19 Johnson WR, Artino Jr AR, Durning SJ. Using the think aloud protocol in health professions education: an interview method for exploring thought processes: AMEE Guide No. 151. Med Teach 2023; 45 (09) 937-948
• 20 Hooper PL. Expert Titration of Multiple Vasoactive Drugs in Post-Cardiac Surgical Patients: An Interpretive Study of Clinical Judgement and Perceptual Acuity. San Francisco:: University of California;; 1995
• 21 Miller K, Mosby D, Capan M. et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
• 22 Faria P. . figma: Web Client/Wrapper to the Figma API. Published online 2023. Accessed May 23, 2024 at: https://github.com/pedropark99/figma , Https://pedropark99.github.io/figma/
• 23 Brooke J. Sus: a “quick and dirty'usability. Usability Eval Ind. 1996; 189 (03) 189-194
• 24 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
• 25 International Organization for Standardization. ISO 9241–210:2019 Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems;. 2019
• 26 Kumah-Crystal Y, Lehmann CU, Albert D. et al. Vanderbilt EHR Voice Assistant (VEVA) supports clinicians. . Appl Clin Inform 2023; 15: 199-203
• 27 Campos TNC, Nogueira JAM, Silva TLS. et al. Development and validation of an application for prevention of sleep disorders in puerperal women. Sleep Sci 2022; 16: 1-70
• 28 Hettinger KN, Snyder ME, Adeoye-Olatunde OA, Russ-Jara AL. Usability evaluation of a community pharmacy health information exchange interface prototype. Appl Clin Inform 2023; 14 (04) 811-821
• 29 Brehaut JC, Graham ID, Wood TJ. et al. Measuring acceptability of clinical decision rules: validation of the Ottawa acceptability of decision rules instrument (OADRI) in four countries. Med Decis Making 2010; 30 (03) 398-408
• 30 de Vos-Kerkhof E, Graham ID, Korfage IJ. et al. Measuring acceptability of clinical decision rules using the Ottawa Acceptability of Decision Rules Instrument (OADRI): a pediatric emergency care study. Int Arch Nurs Health Care 2016; 2: 44
• 31 Jankowski N, Schönijahn L, Salchow C, Ivanova E, Wahl M. User-centred design as an important component of technological development: improving stroke rehabilitation systems with user acceptance and satisfaction surveys. Curr Dir Biomed Eng 2017; 3 (01) 69-73
• 32 Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
• 33 Mihas P. Qualitative data analysis. In: Oxford Research Encyclopedia of Education;. 2019
• 34 QSR International. Nvivo (Version 12.1) qualitative data analysis software. Published online 2018
• 35 Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. Morgan Kaufmann; 2016
• 36 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4 (03) 114-123
• 37 Vlachogianni P, Tselios N. Perceived usability evaluation of educational technology using the System Usability Scale (SUS): a systematic review. J Res Technol Educ 2022; 54 (03) 392-409
• 38 Faller N, Stalder O, Limacher A. et al. Frequency of use and acceptability of clinical prediction rules for pulmonary embolism among Swiss general internal medicine residents. Thromb Res 2017; 160: 9-13
• 39 Trillium Health Partners. Nitroglycerin- Adult Parenteral Therapy Drug Information Order set. Published online 2019
• 40 Marcos M, Maldonado JA, Martínez-Salvador B, Boscá D, Robles M. Interoperability of clinical decision-support systems and electronic health records using archetypes: a case study in clinical trial eligibility. J Biomed Inform 2013; 46 (04) 676-689
• 41 Yoo J, Lee J, Min JY. et al. Development of an interoperable and easily transferable clinical decision support system deployment platform: system design and development study. J Med Internet Res 2022; 24 (07) e37928
• 42 Ash JS, Sittig DF, McMullen CK. et al. Multiple perspectives on clinical decision support: a qualitative study of fifteen clinical and vendor organizations. BMC Med Inform Decis Mak 2015; 15 (01) 35
• 43 Cooper AL, Brown JA, Eccles SP, Cooper N, Albrecht MA. Is nursing and midwifery clinical documentation a burden? An empirical study of perception versus reality. J Clin Nurs 2021; 30 (11-12): 1645-1652
• 44 Tung T-H, DeLaurentis P, Yih Y. Uncovering discrepancies in IV vancomycin infusion records between pump logs and EHR documentation. Appl Clin Inform 2022; 13 (04) 891-900
• 45 Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform 2016; 25 (Suppl. 01) S103-S116
• 46 Dunn Lopez K, Gephart SM, Raszewski R, Sousa V, Shehorn LE, Abraham J. Integrative review of clinical decision support for registered nurses in acute care settings. J Am Med Inform Assoc 2017; 24 (02) 441-450
• 47 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Manage Inf Syst Q 1989; 13: 319-340
• 48 Sagnier C, Loup-Escande E, Lourdeaux D, Thouvenin I, Valléry G. User acceptance of virtual reality: an extended technology acceptance model. Int J Hum Comput Interact 2020; 36 (11) 993-1007
• 49 Ronquillo CE, Peltonen LM, Pruinelli L. et al. Artificial intelligence in nursing: Priorities and opportunities from an international invitational think-tank of the Nursing and Artificial Intelligence Leadership Collaborative. J Adv Nurs 2021; 77 (09) 3707-3717
• 50 Pal D, Vanijja V. Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India. Child Youth Serv Rev 2020; 119: 105535
• 51 Silva P. Davis' technology acceptance model (TAM)(1989). Information Seeking Behavior and Technology Adoption: Theories and Trends. IGI Global; 2015; 205-219
• 52 Khong PCB, Hoi SY, Holroyd E, Wang W. Nurses' clinical decision making on adopting a wound clinical decision support system. Comput Inform Nurs 2015; 33 (07) 295-305
• 53 Dhopeshwarkar RV, Freij M, Callaham M. et al. Lessons learned from a national initiative promoting publicly available standards-based clinical decision support. Appl Clin Inform 2023; 14 (03) 566-574
• 54 Schwartz JM, George M, Rossetti SC. et al. Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study. JMIR Human Factors 2022; 9 (02) e33960
• 55 Kwong JCC, Erdman L, Khondker A. et al. The silent trial - the bridge between bench-to-bedside clinical AI applications. Front Digit Health 2022; 4: 929508

Publication History

Received: 14 December 2023

Accepted: 14 May 2024

Article published online:
24 July 2024