Design and Usability of an Electronic Health Record—Integrated, Point-of-Care, Clinical Decision Support Tool for Modeling and Simulation of Antihemophilic Factors

Abstract

Background With the consequences of inadequate dosing ranging from increased bleeding risk to excessive drug costs and undesirable administration regimens, the antihemophilic factors are uniquely suited to dose individualization. However, existing options for individualization are limited and exist outside the flow of care. We developed clinical decision support (CDS) software that is integrated with our electronic health record (EHR) and designed to streamline the process for our hematology providers.

Objectives The aim of this study is to develop and examine the usability of a CDS tool for antihemophilic factor dose individualization.

Methods Our development strategy was based on the features associated with successful CDS tools and driven by a formal requirements analysis. The back-end code was based on algorithms developed for manual individualization and unit tested with 23,000 simulated patient profiles created from the range of patient-derived pharmacokinetic parameter estimates defined in children and adults. A 296-item heuristic checklist was used to guide design of the front-end user interface. Content experts and end-users were recruited to participate in traditional usability testing under an institutional review board approved protocol.

Results CDS software was developed to systematically walk the point-of-care clinician through dose individualization after seamlessly importing the requisite patient data from the EHR. Classical and population pharmacokinetic approaches were incorporated with clearly displayed estimates of reliability and uncertainty. Users can perform simulations for prophylaxis and acute bleeds by providing two of four therapeutic targets. Testers were highly satisfied with our CDS and quickly became proficient with the tool.

Conclusion With early and broad stakeholder engagement, we developed a CDS tool for hematology provider that affords seamless transition from patient assessment, to pharmacokinetic modeling and simulation, and subsequent dose selection.

Keywords

software design - therapeutic drug monitoring - usability testing - factor VII - factor VIII - factor IX

Background and Significance

The large degree variability in dose-exposure relationships between patients contributes to uncertainty in our ability to predict whether a given patient at a given dose of a given drug will respond favorably, develop unwanted side effects, or perceive the requirements for administration cumbersome enough to interfere with their daily quality of life (QOL). Antihemophilic factors represent one-such class; dose-exposure profiles vary between and within patients (as they grow and develop) and the consequences of inadequate dosing are marked. Too little factor increases the risk of serious bleeds while too much factor results in excessive drug costs and, in some cases, undesirable administration regimens. For these drugs, clinicians can apply specialized knowledge of pharmacokinetics (PK) to account for this variability and individualize dosing recommendations. PK-guided dose individualization affords the opportunity to achieve clinician-driven therapeutic targets with no more factor than is necessary to prevent hemorrhagic episodes.[1] In fact, PK strategies have been used to guide the replacement of various clotting factors for more than two decades.[2] [3] [4] [5]

Successful application of PK-guided dose individualization to routine patient care requires (1) comprehensive knowledge of the patient, (2) a thorough understanding of the pharmacologic principles that drive the relationship between dose-exposure, and (3) expertise in mathematical and PK modeling and simulation.[6] Providers with expert knowledge of the latter often have limited involvement with the patient or their primary medical team and engaging them outside the normal flow of care can introduce delays in arriving at the best dosing solution for the patient.[7] For decades, countless computer-based tools have been developed in an attempt to integrate PK-individualization into the clinical care workflow.[8] [9] [10] [11] Most exist as standalone applications and relatively few were designed to interface with the electronic health record (EHR).[12] [13] Consequently, very few of these tools experience utilization that extends beyond the academic center at which they are developed. Moreover, examinations of usability (when conducted) often engage individuals who already possess expertise in PK rather than clinician end users for whom they are intended.[12] [13] As a result, valid estimates of usefulness remain undefined.

Objective

At our institution, we employ PK individualization for difficult to manage hemophilia patients who are unresponsive to standard dosing regimens. This is routinely accomplished using a classical pharmacokinetic approach where multiple (often five or more), patient-specific, exposure levels are examined to describe a mathematical model with parameter estimates unique to the individual patient that can be used for dose simulation. As described above, this typically occurs outside the flow of care via consultation with clinical pharmacology experts. Our primary goal was to create a simple, tailored, clinical decision support (CDS) solution that addresses our providers needs and integrates into the flow of care. In recent years, there has also been increasing dialogue related to the value of population PK (popPK) for dose optimization where a limited number of exposure levels (typically 1–2) are used to modify parameter estimates that derive from a broader population of individuals. Our secondary goal was to provide a solution that allows clinicians to gain additional experience with the predictive performance of these models. The objective of this paper is to describe the development and usability of this CDS tool for antihemophilic factor dose individualization.

Methods

Design

Our approach followed a typical user-centered design process to ensure the successful development of a product that targets the needs of its users.[14] [15] This included the identification of users and an early assessment of workflows and clinical challenges. User stories were constructed to ensure that the project team developed a solution which addressed the needs of the providers impacted by the application. Construction of the wireframe, or high-fidelity mockup, was followed by iterative prototyping and empirical measurement throughout design and development. The development strategy was driven by the features associated with successful CDS namely: (1) involving local users, (2) integrating with the charting/order entry system, (3) avoiding additional data entry, (4) availability at the time/location of decision making, (5) providing a recommendation (vs. an assessment), and (6) justifying the decision with evidence.[16]

Requirements Analysis

The project began by engaging physicians, nurses, pharmacists, and pharmacologists at a standalone pediatric hospital that serves pediatric patients and provides consultative services for adult patients at a neighboring institution. The existing processes associated with antihemophilic factor PK individualization were documented including the sequence of tasks and information flows. High-level tasks were deconstructed into subtasks and operations, and the information sources required in each subtask were recorded. The output included process charts, task flow diagrams, task decomposition tables, and use case scenarios.[17] Additional discussions centered on functionality currently not embedded in the workflow but desirable for optimal CDS utility. The major features identified by the users included: (1) transparent integration with the EHR, (2) application to factors VII, VIII, and IX, (3) application to standard and extended half-life products, (4) simulation for both prophylactic and presurgical/acute bleed regimens, and (5) integration of both classical and popPK models.

User Stories and Iteration Methodology

Features of the application were designed and implemented in an iterative fashion following agile development principles.[18] Four clinical scenarios were used to frame the user stories:

1. AV had a factor 9 pharmacokinetic profile evaluated at 7 years and 7 months of age. You would like to identify a prophylactic dose that results in a trough activity level of 5%.

2. FL had a factor 9 pharmacokinetic profile evaluated at 1 year and 4 months of age. He will be coming in for surgery and you would like to identify a loading and maintenance dose for acute bleeds that maintain peak levels around 80% and trough levels around 30%.

3. PP had a factor 7 pharmacokinetic profile evaluated at 42 years of age. You would like to explore whether a more abbreviated sampling strategy could have still reliably predicted PP's pharmacokinetic profile.

4. LF had a full factor 8 pharmacokinetic profile evaluated at 3 years of age and had repeated random sampling 5 months later. You would like to identify how effectively existing population models predict these levels.

These user stories were integral to the development and quality assurance (QA) process wherein stakeholders worked with a business and QA analyst to document, justify, and prioritize feature development and bug fixes. A standardized format for user stories was implemented, significantly reducing confusion and development/debug cycles when compared against prior software development.[19] User stories helped prioritize work and prompted in-depth discussions with stakeholders that aided the development team's understanding of the business cases, justifications behind features, and bug fix requests. DevOps process, the practice of development and operations participating together in the entire service lifecycle from design through the development process to production support, was followed as a part of the product service lifecycle.

Front-End Design

Information gathered from the requirements analysis informed the design of the CDS prototype which was developed in Adobe XD. Highcharts, a javascript library, was used to implement visualizations presented by the user interface (UI). The design was guided by feedback gathered from the development of an earlier CDS tool developed at our institution and a 296-item heuristic checklist.[19] [20] The major domains under consideration were: visibility of system status, match between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design, and help and documentation.

Responsive web design best practices were used to implement the look and feel of the web application and achieve a consistent experience across different web browsers.[21] Angular bootstrap was used for the various UI widgets which were further customized based on the required functionality.[22] [23] The UI interface was created with a strong focus on reusability. As a result, the web application was implemented as a single page application using Angular 6. Visual Studio was used to integrate back-end code with the front-end UI.

Back-End Development

Owing to the historical reliance on, and comfort of our users with, classical PK approaches, dose individualization in this application occurs with the use of patient-specific PK parameter estimates. PopPK models were also integrated into the application to facilitate practice improvement considerations. Though not used to support dose individualization with the initial release of the application, providers can explore whether publicly accessible popPK models effectively describe the disposition of antihemophilic factor in their patient population. When our providers have accumulated sufficient evidence to support the use of specific popPK models, the provider team can request that one or more of these models be pulled forward to permit dose optimization using a minimal sampling approach coupled with popPK.

The software flow was based on an algorithm developed and modified by the clinical pharmacologist during the course of practice. For application of classical PK, we formulated our compartmental modeling approach as a nonlinear least square problem, with parameters estimated by curve-stripping and optimized using the Levenberg–Marquardt's algorithm.[24] A series of nested logic functions were used to determine whether the data should be fit to a one-, two-, or three-compartment model. For datasets where a prestudy dose contributed to measurable factor activity levels, the baseline contribution of the prior dose was removed from the profile using the superposition principle.[25] For sparsely sampled datasets, a decision tree aided in model selection or defaulted to the application of popPK models. Twelve peer-reviewed popPK models, for which all parameters were clearly specified in the publication or provided to the authors via personal communication, were coded into the application ([Supplementary Material], available in the online version). PopPK models were accessible to the user only if all of the relevant covariates were populated in the EHR.

The simulation engine was coded to individualize dosing for prophylaxis or acute bleeds using algorithms that require the user to specify two inputs (i.e., dose, dosing interval, desired peak, and desired trough) with the engine returning the remaining two as outputs. Additional features were added for simulating nonuniform dosing intervals (e.g., Monday, Wednesday, and Friday dosing). Quantitative goodness-of-fit (GOF) criteria, traditionally used to determine the appropriateness of a model (e.g., objective function, weighted sum of squares, and coefficients of variation), were log-adjusted, weighted, and combined into a 10-point scale to inform a visual indicator that alerts the user to the strength of the model.

Unit testing for each executable path was performed with 23,000 simulated patient profiles. These profiles were created from the range of patient-derived pharmacokinetic parameter estimates observed in children and adults with the commercially available factor formulations. Iterative refinement of the algorithm was performed until no fewer than 98% of results from the simulated profiles were in agreement with the expected results generated by applying the same calculations in Excel. Recognizing that we are unable to fully eliminate edge cases, a penalty structure was nested into the GOF algorithm to inform the user when the reliability of the model may be compromised. The algorithm was implemented using C# programming language. Jupyter Notebook was used for visual evaluation.

Integration with the Electronic Health Record

To enable interoperability with any EHR system, we created an Application Programming Interface (API) for EHR integration. We designed an industry standard REST API, which delivers EHR data that our tool can consume provided that the output complies with a predefined signature.[26] For example, patient demographics are currently being provided by a REST API which requires a medical record number (MRN) and in return it sends patient name and DOB among other demographics. With this architecture, the application can consume demographics from any REST API that complies with the predefined API signature for demographics features. If the application owner decides to switch EHR vendors, they can simply create a REST API which complies with defined signature and point the application to that API. These REST APIs are generally not provided by EHR vendors. Application owners need to create REST APIs, or alternatively FHIR APIs, which can communicate with the EHR and supply the requisite data for the application. Our decision to rely on a REST API was influenced by the limited availability of FHIR services at the time of development.

For our system (Cerner Millenium, Cerner Corporation, Kansas City, Missouri, United States), we also created a PowerForm (i.e., a template for clinical data entry) to facilitate data requests and eliminate duplicative data capture. Completing the form indicates that the event is ready to be pulled into our application. To access EHR-based data, users must be registered in a prespecified authentication group controlled by our security team. As an added QA measure, the application records user information and actions when EHR data are requested. The software was also built with a manual entry feature to accommodate the analysis of data from patients who are not contained within the EHR.

Usability Testing

Structured cognitive walkthroughs (CW) were conducted with content experts (CE) and end-users (EU) representing medicine, pharmacy, pharmacology, and nursing to assess efficiency, ease-of-use, and user satisfaction among a representative user populations. CE reflected individuals having completed formal postdoctoral training in clinical pharmacology. EU reflected clinicians that could reasonably be expected to interact with the software (i.e., physicians, nurses, and pharmacists). Using the application, participants were asked to complete a series of tasks centered on the four clinical scenarios described above. All four scenarios were of similar complexity and completed in one sitting lasting approximately 1 hour. We employed a think-aloud protocol in which participants were encouraged to verbalize their thoughts while audio and video of their interaction with the application were captured on logging software (Techsmith Morae, Okemos, Michigan, United States). We also recorded time to complete tasks, number of clicks required to complete tasks, task success, and user perceptions of task ease/difficulty. The Post-Study System Usability Questionnaire (PSSUQ v3) was used to capture overall user satisfaction and perceived usability.[27] Recommendations arising from the CW were incorporated into the application's final design.

Data Analysis

Standard descriptive statistics were applied to describe the participant populations and the usability outcomes. To identify whether task times improved with repeated use of the application, differences in time to completion between comparable, successive tasks were analyzed using a paired t-test or analysis of variance. Statistical differences in task performance between user subpopulations were examined by application of a two-tailed, unpaired Student's t-test. The significance limit accepted for all statistical analyses was α = 0.05. All analyses were performed in SPSS v24 (IBM Corp, Armonk, New York, United States).

Results

Form and Function

The CDS tool developed systematically walks the EU through a series of five to seven screens:

Patient search ([Fig. 1]): This screen allows users to enter an MRN or select manual entry if the patient is not in our EHR system (e.g., patients receiving care at the neighboring adult hospital). Users that are not on the hospital network will encounter an initial authentication screen prior to this patient search page.

Fig. 1 Patient search screen where users can enter a medical record number (upper) and identify the pharmacokinetics study of interest (lower) (fictional patient data). Manual data can also be selected subsequently bypassing the search results screen.

Search results ([Fig. 1]): Entering an MRN will pull up all instances of antihemophilic factor PK data available for that child on the search results page. From this page users can select the PK study date of interest or return to patient search. Selecting “manual entry” on the previous page will bypass this screen.

Laboratory results ([Fig. 2]): This screen displays the relevant patient information, dosing history, and laboratory information that have been pulled from the EHR for confirmation and visual inspection. Users can deselect data points that appear errant using the exclude laboratory results checkboxes or “switch to manual entry” mode if information is incorrect or missing so that the data can be entered or corrected. Alternatively, users can proceed directly to curve fitting via the “view model fit” button. Users with sparse data may encounter one of two warnings alerting them to issues of reliability or advising them of analysis types to which they are restricted.

Fig. 2 Laboratory results page where users can confirm the accuracy of the data being imported (upper) and transition to the model fit page where they can examine modeling results and explore relevant popPK models (lower) (fictional patient data).

Model fit ([Fig. 2]): After executing the modeling algorithm, users observe a graphic of the curve fit detailing the model-type. They can assess the GOF by looking at the relationship between the observed data and the curve on the graph, by examining the observed and predicted concentrations in the table to the left of the graph or via the tachometer which consolidates GOF criteria into a single indicator (green alerts the user that the model is strong, yellow signals caution, and red indicates that there may be a problem with the fitting). If users are dissatisfied with the model, they can go back to laboratory results and reinspect the raw data to determine whether selected samples need to be excluded from the fit. Users that are satisfied with the model can proceed to “simulation” or stay on this page to examine how well-published popPK models fit their patient's data. Only popPK models for which all of the necessary patient specific information is available are presented for the user. “Switch to manual entry” mode on the preceding screen allows the user to update any missing information and access the omitted popPK models.

Simulation ([Fig. 3]): Users can explore dosing recommendations with user-defined target exposures or examine exposures that result from user-defined dosing regimens. Users are not restricted to regularly scheduled administration and can simulate irregular dosing intervals that may be more convenient for their patients. The simulations can be repeated ad infinitum until the clinician settles on a dosing regimen that achieves the goals of therapy.

Fig. 3 Simulation page where users can conduct simulations for target dose, interval, peak, or trough for prophylaxis (upper) and acute bleeds (lower) (fictional patient data).

View report: When users have decided on a final dosing recommendation, they can click “create report,” review their recommendations, add comments, hide or expose popPK curve fits, and download the report.

Usability

In total, 12 CE and 12 EU completed the usability testing. Their baseline characteristics are detailed in ([Table 1]). The majority of these individuals were under the age of 40 and had been in practice less than 10 years. There was a slight preponderance of females over males, which is consistent with the demographic breakdown of the workforce at our pediatric institution. With one exception, all of these participants spent more than 15 hours on a computer each week, most commonly a Windows based-operating system, and accessed the EHR daily.

Table 1 Characteristics of participants in the usability testing

Characteristic	Group	Content experts (n = 12)	End users (n = 12)
Gender	Female	6	9
	Male	6	3
Age group (y)	26–39	8	9
	40–59	3	2
	60–74	1	1
Current role	Physician	8	6
	Pharmacist	3	3
	Nurse	1	2
	Other	0	1
Years in current role	<5	3	5
	5–10	6	3
	10–15	0	2
	>15	3	2
Activities performed on computer	Median (range)	5 (3–8)	5.5 (2–8)
Hours/week on computer	5–15	1	0
	15–25	4	4
	26±	7	8
Computer platform most often used	Mac	2	3
	Windows	10	9
Browsers used when computing	Median (range)	1 (1–3)	2 (2–4)
Frequency with which EHR is accessed	Daily	8	11
	Weekly	1	0
	Never	3	1
Frequency with which computerized CDS tools are used	Daily	1	0
	Once or twice a week	2	1
	About once a month	1	3
	A couple of times	4	5
	Never	4	3
Frequency with which TDM is used in clinical decision making	Daily	0	1
	Once or twice a week	3	1
	About once a month	2	6
	A couple of times	1	1
	Never	6	1
Frequency with which PK calculations are applied to patient care	Once or twice a week	1	1
	About once a month	2	1
	A couple of times	2	4
	Never	7	6
Proficiency with PK calculations	Strong	3	0
	Moderate	8	3
	Weak	1	6
	Not proficient at all	0	3

Abbreviations: CDS, clinical decision support; EHR, electronic health record; PK, pharmacokinetics; TDM, therapeutic drug monitoring.

Usability testing revealed that our CDS tool could be efficiently navigated by our testers. Median task time across all tasks was 13.2 seconds with a median 1.0 mouse clicks per task. Greater than 95% of tasks were completed with no difficulty. For comparable tasks that were encountered more than once during testing, task times dropped significantly for all but one that asked the user to identify the newly simulated dose ([Table 2]). After experience was gained with the application, users spent most of their time simulating new dosing regimens and finalizing the report for download. By contrast, observational or one-click task such as performing curve fitting and identifying the mathematical model required the least amount of time.

Table 2 Time to complete selected tasks. Data are represented as mean ± standard deviation

Task	Event 1 (min)	Event 2 (min)	Event 3 (min)	Event 4 (min)	p-Value
Import data[a]	0.72 ± 0.40	0.27 ± 0.18	0.20 ± 0.09	0.31 ± 0.20	<0.01
Inspect the data[a]	0.66 ± 0.41	0.32 ± 0.21	0.24 ± 0.15	0.19 ± 0.10	<0.01
Perform curve fitting[a]	0.34 ± 0.28	0.06 ± 0.04	0.05 ± 0.02	0.07 ± 0.09	<0.01
Evaluate the goodness of fit[b]	0.50 ± 0.33	0.17 ± 0.10	0.18 ± 0.16		<0.01
Identify the model that was fit[b]	0.43 ± 0.53	0.09 ± 0.22	0.03 ± 0.01		<0.01
Simulate a new dose for specified therapeutic targets[b]	0.86 ± 0.26	0.45 ± 0.20	0.34 ± 0.12		<0.01
Identify the new dose	0.17 ± 0.34	0.22 ± 0.17	0.20 ± 0.11		0.694
Modify the simulation to adjust the dose	0.85 ± 0.51	0.33 ± 0.14			<0.01
Examine the therapeutic targets with the modified dose	0.36 ± 0.23	0.25 ± 0.13			0.025
Finalize the report[c]	1.22 ± 0.56	0.98 ± 0.46	0.30 ± 0.18		<0.01

^a Events 2, 3, and 4 significantly faster than event 1.

^b Event 2 and 3 significantly faster than event 1.

^c Event 3 significantly faster than events 1 and 2.

No differences in mean or total task times were observed between EU and CE. Similarly, performance metrics were largely uniform across our population irrespective of their background characteristics. No differences were observed by gender, age group, general computer use patterns, training, or tenure. The number of clicks required to navigate through the cases did decrease systematically with increased frequency of therapeutic drug monitoring experience, but this did not attain statistical significance (p = 0.054).

When observation and feedback obtained from usability testing was collated, issues surrounding five common themes emerged: (1) design: esthetic preferences related to color use, font size, and positioning of various elements within the application represented the most abundant comments we received (34%). (2) Feedback: confusion around the feedback provided by the application represented the next most abundant issues identified by users (29%). In this category, we include descriptors (e.g., headers, labels, figure legends, and units) that were inadequate, nonsensical, or missing. (3) Features: issues specific to selected features of the application represented 21% of the feedback received. These included pop-up notifications that the user could not close, graphical features that autoscaled to data, user selections that were reflected on a graph but not in the companion table, and visibility of the help feature which could not be identified by eight users. (4) System: issues stemming from the development mode in which we were working contributed to approximately 12% of issues. Some users experienced a system that was operating slowly, freezing on selected screens, or jumping screens unbeknownst to the user. There were also episodes of screens prepopulating with cached information from the preceding clinical scenario. (5) Navigation: less than 5% of the issues identified related to navigation within the application and all of these could be attributed to the user navigating to an earlier screen using the back button in the browser rather than the back button nested in the application. Irrespective of these issues, satisfaction with the application was extremely positive. Using a 7-point Likert scale with 1 representing the most favorable response and 7 the least favorable response, overall satisfaction with the application rated a 1.42. Scores for system quality rated 1.56, information quality rated 1.73, and interface quality rated 1.33. The distribution of scores for each element of the PSSUQ is provided in [Fig. 4].

Fig. 4 Histogram of Poststudy System Usability Questionnaire responses provided by our usability testers.

Discussion

Main Findings

In this manuscript, we describe the development and testing of CDS software that shifts the flow of PK-based therapeutic decision-making from a consultative service model to seamless point-of-care execution. Direct access to antihemophilic factor modeling and simulation, 24/7, offers primary hematology providers enhanced flexibility when caring for patients. Though clinical pharmacologists remain accessible to the hematology team, they satisfy an ancillary role reserved for mathematically complex scenarios. This approach diverges from traditional hospital-based PK services where specialists are consulted to conduct modeling and simulation. However, we made a business decision to prioritize the efficiency of care delivered by the patient's primary provider over the protection of a billable domain for a secondary specialist.

We focused on developing an application that integrates transparently with the EHR, intelligently filtering, organizing, and delivering the relevant information to the tool at the appropriate time.[13] We also vetted the application with a multidisciplinary group of providers to ensure that it performs optimally and supports the needs of the end user. To that end, we appear to have been successful as evidenced by median scores from the usability questionnaires that lie between 1 and 2 and objective performance metrics which demonstrate that key tasks are rapidly learned after working through one use case.

Related Dosing Tools

Two other publicly accessible PK-driven dosing tools have been described, though there are notable differences with the tool we developed. The U.S. Food and Drug Administration (FDA)-cleared myPKFiT is a web-based dosing tool for optimization of Advate therapy.[28] WAPPS-Hemo is a web-based service that can be applied to factor VIII and IX products.[29] Both adopt a popPK approach though only myPKFiT references their base model, which has been peer-reviewed and published. Both must be accessed independently and rely on the transfer or transcription of patient specific variables. As a service, WAPPS-Hemo also introduces a delay between the time the data are submitted, and a dosing recommendation is returned as compared with our tool which delivers a recommendation in real-time as the clinician performs the simulations. Finally, each has constraints with respect to the execution of their modeling and/or simulation algorithms.[30]

Importantly, side-by-side comparisons of these two tools demonstrates that they do not always arrive at the same dose and both generate a different dose than is derived using traditional popPK software (NONMEM).[30] Whether this results from the uncertainty nested in their parameter estimates, or the fact that they may be applying a model developed in a population with characteristics that differ from the patients that were examined is unclear. Armed with these data, our providers recognized the need to collect more empiric data on the performance of various population models before broad implementation which is why our tool relies on classical PK for dose simulation but integrates popPK at the modeling step to provide the user with estimates of reliability and uncertainty that can be expected with these approaches.

Limitations

There exist a several of limitations tied to the development of the application. At present, the application represents a partially rather than fully EHR-integrated CDS tool. Single sign-on authentication onto the institutional computing system permits access to the application via a weblink, as opposed to direct access from within the EHR, necessitating the user to enter the patients MRN. We are exploring the ability to embed the weblink directly into the PowerForm, thus bypassing the need for MRN entry but have yet to test this functionality. We also considered, but elected to delay, an automated push of the results back into the EHR. As with our previous CDS tools, we commit a period of time to fully QA the application's output in a real-world setting before automating delivery back to the medical record. When sufficient experience has been gained, a routine HL7 protocol can be used to implement this change and push the output to a dedicated location in the patients' chart.

Additional limitations relate to the findings of our usability testing. All testers in this study derived from a single institution with a robust clinical pharmacology program. As a result, they may share a more complete understanding of PK than users at other institutions. This may also explain why we observed essentially no differences in performance metrics between our CE and end users. Our inability to discriminate differences in performance as a function of participant characteristics may have been influenced by sample size. However, our previous usability study for a different PK-based CDS application with a similar sample size could still to distinguish differences between different participant groups.[19] Alternatively, this might be explained by the slightly more complex nature of the modeling and simulation requirements for the medications addressed by this application. This assertion is corroborated by slightly longer task times for similar actions in this, compared with the earlier, study.

Next Steps

Immediate next-steps involve prospectively examining the accuracy and predictive performance of the CDS to expose whether the back-end model can be augmented. We are also architecting a companion patient-facing application. Despite an established link between adherence, frequency of bleeds, and QOL,[31] [32] [33] adherence rates vary substantially and strategies to promote adherence in hemophilia patients are not widely addressed.[32] [33] [34] [35] [36] [37] [38] Patients rarely receive more than superficial insight into the dose selection strategies used by their prescriber or an evidence-based explanation detailing the consequences of altered dosing regimens. Consequently, it is unrealistic to expect that patients with chronic conditions are self-motivated to comply with regimens that are incompatible with their lifestyle.[39] [40] [41] Our application feature allowing providers to explore dosing recommendations beyond the idealized (e.g., administration Monday and Thursday vs. every 72 hours) was borne out of this understanding; however, the simple act of individualizing treatment cannot guarantee adherence. Providing access to adaptive, individualized, patient-centered education may have the power to enhance knowledge around prescribed dosing regimens, promote shared decision making, and ultimately influence adherence behaviors.

Conclusion

We developed a user-friendly CDS tool for hematology provider that affords seamless transition from patient assessment, to pharmacokinetic modeling and simulation, and subsequent dose selection.

Clinical Relevance Statement

Coupling early and broad stakeholder engagement with a provider-centered design strategy can result in point-of-care solutions that are favorably received by clinicians. EHR integration further increases the utility of these solutions and the likelihood of uptake in clinical settings.

Multiple Choice Questions

1. Successful clinical decision support tools possess which of the following attributes?

   • They exist as standalone solutions.

   • They require additional data entry.

   • They are available at the time/location of decision-making.

   • They provide an assessment only without a formal recommendation.

   Correct Answer: The correct answer is option c.

2. Which of the following statements is true about structured CW?

   • They should only be conducted with CE.

   • They should only be conducted with EU.

   • They only generate subjective data on interactions between the user and your software.

   • They generate data that can be used to iterate your software design.

   Correct Answer: The correct answer is option d.

Conflict of Interest

None declared.

Acknowledgments

We sincerely appreciate the expert contributions of Cathy Jackson, Hope Davis, Jenni Wilkerson, Earl Glynn, Brian Rivera, and Sara Winkle. Funding for the development of this software was provided by generous gifts from Giannini Fund and the Stowers Family Foundation.

Authors' Contributions

S.M.A.R. conceived of the application, developed the initial algorithm around which the software was based, led the development of the decision support tool, and conducted the usability testing. S.L.C. and B.W. coordinated the requirements analysis for the clinical aspects of this tool. H.G., P.G., and A.K. were involved in coding of the back-end analytics. S.M.A.R., H.G., and P.G. undertook unit testing and validation. M.B. and A.G. were responsible for the design and coding of the UI. H.G. was responsible for integration of the software with the electronic health record. A.K. was responsible for supervising all informatics activities. All authors reviewed and approved the manuscript.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. All participants were enrolled with informed consent under a protocol that was reviewed and approved by the Institutional Review Board at Children's Mercy Hospital (IRB# 00000285).

• References

• 1 Carlsson M, Björkman S, Berntorp E. Multidose pharmacokinetics of factor IX: implications for dosing in prophylaxis. Haemophilia 1998; 4 (02) 83-88
• 2 Blowey DL, Kearns GL. Congenital nephrotic syndrome alters antithrombin-III disposition in children. Clin Pharmacol Ther 1998; 63: 76
• 3 Shapiro AD, Korth-Bradley J, Poon MC. Use of pharmacokinetics in the coagulation factor treatment of patients with haemophilia. Haemophilia 2005; 11 (06) 571-582
• 4 Shakhnovich V, Daniel J, Wicklund B, Kearns G, Neville K. Use of pharmacokinetic modelling to individualize FFP dosing in factor V deficiency. Haemophilia 2013; 19 (02) 251-255
• 5 Brown JT, Wicklund BM, Abdel-Rahman SM. Individualized factor IX dosing: application of pharmacokinetic modeling to optimize pharmacodynamic activity. Haemophilia 2015; 21: e125-e127
• 6 Durieux P, Trinquart L, Colombet I. , et al. Computerized advice on drug dosage to improve prescribing practice. Cochrane Database Syst Rev 2008; 3 (03) CD002894
• 7 Buclin T, Gotta V, Fuchs A, Widmer N, Aronson J. Monitoring drug therapy. Br J Clin Pharmacol 2012; 73 (06) 917-923
• 8 Sheiner LB, Rosenberg B, Melmon KL. Modelling of individual pharmacokinetics for computer-aided drug dosage. Comput Biomed Res 1972; 5 (05) 411-459
• 9 Proost JH, Meijer DK. MW/Pharm, an integrated software package for drug dosage regimen calculation and therapeutic drug monitoring. Comput Biol Med 1992; 22 (03) 155-163
• 10 Buffington DE, Lampasona V, Chandler MHH. Computers in pharmacokinetics. Choosing software for clinical decision making. Clin Pharmacokinet 1993; 25 (03) 205-216
• 11 Lacarelle B, Pisano P, Gauthier T. , et al. Abbott PKS system: a new version for applied pharmacokinetics including Bayesian estimation. Int J Biomed Comput 1994; 36 (1-2): 127-130
• 12 Nieuwlaat R, Connolly SJ, Mackay JA. , et al; CCDSS Systematic Review Team. Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review. Implement Sci 2011; 6: 90
• 13 Fuchs A, Csajka C, Thoma Y, Buclin T, Widmer N. Benchmarking therapeutic drug monitoring software: a review of available computer tools. Clin Pharmacokinet 2013; 52 (01) 9-22
• 14 Wallach D, Scholz SC. User-Centered Design: Why and How to Put Users First in Software Development. In Software for People. Heidelberg, Germany: Springer Berlin Heidelberg; 2012: 11-38
• 15 Gould JD, Lewis C. Designing for usability: key principles and what designers think. Commun ACM 1985; 28: 300-311
• 16 Lobach D, Sanders GD, Bright TJ. , et al. Enabling Health Care Decision making Through Clinical Decision Support and Knowledge Management. Evidence Report No. 203. (Prepared by the Duke Evidence-based Practice Center under Contract No. 290–2007–10066-I.) AHRQ Publication No. 12–E001-EF. Rockville, MD:Agency for Healthcare Research and Quality. Evid Rep Technol Assess (Full Rep) 2012; 203: 1-784
• 17 Wiegers K, Beatty J. Software Requirements. 3rd ed. Redmond, Washington: Microsoft Press; 2013
• 18 Highsmith J, Cockburn A. Agile Software Development: the business of innovation. IEEE Computer 2001; 34: 120-127
• 19 Abdel-Rahman SM, Breitkreutz ML, Bi C. , et al. Design and testing of an EHR-integrated, busulfan pharmacokinetic decision support tool for the point-of-care clinician. Front Pharmacol 2016; 7: 65
• 20 Weiss E. Making Computers People-Literate. San Francisco: Jossey-Bass Publishers; 1993
• 21 LePage P. Responsive web design basics. Available at: https://developers.google.com/web/fundamentals/design-and-ux/responsive . Accessed January 16, 2020
• 22 Angular. Available at: https://angular.io/ . Accessed January 16, 2020
• 23 Bootstrap. Available at: https://getbootstrap.com/ . Accessed January 16, 2020
• 24 Gavin HP. The Levenberg-Marquardt algorithm for nonlinear least squares curve-fitting problems (August 3, 2019). Available at: http://people.duke.edu/~hpgavin/ce281/lm.pdf . Accessed January 9, 2019
• 25 Thron CD. Linearity and superposition in pharmacokinetics. Pharmacol Rev 1974; 26 (01) 3-31
• 26 What Is REST. Available at: https://restfulapi.net/ . Accessed January 16, 2020)
• 27 Sauro J, Lewis JR. Quantifying the User Experience Practical Statistics for User Research. Waltham, MA: Morgan Kaufmann; 2012
• 28 myPKFiT for ADVATE. 510(k) Number BK170028. Decision date 12/14/17. Available at: https://www.fda.gov/vaccines-blood-biologics/substantially-equivalent-510k-device-information/bk170028-mypkfit-advate . Accessed May 25, 2019
• 29 Iorio A, Keepanasseril A, Foster G. , et al; WAPPS-Hemo co-investigator network. Development of a web-accessible population pharmacokinetic service-hemophilia (WAPPS-Hemo): study protocol. JMIR Res Protoc 2016; 5 (04) e239
• 30 Preijers T, van Moort I, Fijnvandraat K, Leebeek FWG, Cnossen MH, Mathôt RAA. ; ‘OPTI-CLOT’ Study Group. Cross-evaluation of pharmacokinetic-guided dosing tools for factor VIII. Thromb Haemost 2018; 118 (03) 514-525
• 31 Manco-Johnson MJ, Abshire TC, Brown D. Randomized, controlled, multi-year study to evaluate joint outcomes in young children using recombinant factor VIII (Kogenate® FS). Haemophilia 2006; 12: 17-18
• 32 du Treil S, Rice J, Leissinger CA. Quantifying adherence to treatment and its relationship to quality of life in a well-characterized haemophilia population. Haemophilia 2007; 13 (05) 493-501
• 33 García-Dasí M, Aznar JA, Jiménez-Yuste V. , et al. Adherence to prophylaxis and quality of life in children and adolescents with severe haemophilia A. Haemophilia 2015; 21 (04) 458-464
• 34 Hacker MR, Geraghty S, Manco-Johnson M. Barriers to compliance with prophylaxis therapy in haemophilia. Haemophilia 2001; 7 (04) 392-396
• 35 Lindvall K, Colstrup L, Wollter IM. , et al. Compliance with treatment and understanding of own disease in patients with severe and moderate haemophilia. Haemophilia 2006; 12 (01) 47-51
• 36 Duncan N, Kronenberger W, Roberson C, Shapiro A. VERITAS-Pro: a new measure of adherence to prophylactic regimens in haemophilia. Haemophilia 2010; 16 (02) 247-255
• 37 De Moerloose P, Urbancik W, Van Den Berg HM, Richards M. A survey of adherence to haemophilia therapy in six European countries: results and recommendations. Haemophilia 2008; 14 (05) 931-938
• 38 Armstrong EP, Malone DC, Krishnan S, Wessler MJ. Adherence to clotting factors among persons with hemophilia A or B. Hematology 2015; 20 (03) 148-153
• 39 Razzaboni E, Toss A, Cortesi L. , et al. Acceptability and adherence in a chemoprevention trial among women at increased risk for breast cancer attending the Modena Familial Breast and Ovarian Cancer Center (Italy). Breast J 2013; 19 (01) 10-21
• 40 Will JC, Zhang Z, Ritchey MD, Loustalot F. Medication adherence and incident preventable hospitalizations for hypertension. Am J Prev Med 2015; 50 (04) 489-499
• 41 Liu AY, Hessol NA, Vittinghoff E. , et al. Medication adherence among men who have sex with men at risk for HIV infection in the United States: implications for pre-exposure prophylaxis implementation. AIDS Patient Care STDS 2014; 28 (12) 622-627