Assessing Data Adequacy for High Blood Pressure Clinical Decision Support: A Quantitative Analysis

Abstract

Objective This study examines guideline-based high blood pressure (HBP) and hypertension recommendations and evaluates the suitability and adequacy of the data and logic required for a Fast Healthcare Interoperable Resources (FHIR)-based, patient-facing clinical decision support (CDS) HBP application. HBP is a major predictor of adverse health events, including stroke, myocardial infarction, and kidney disease. Multiple guidelines recommend interventions to lower blood pressure, but implementation requires patient-centered approaches, including patient-facing CDS tools.

Methods We defined concept sets needed to measure adherence to 71 recommendations drawn from eight HBP guidelines. We measured data quality for these concepts for two cohorts (HBP screening and HBP diagnosed) from electronic health record (EHR) data, including four use cases (screening, nonpharmacologic interventions, pharmacologic interventions, and adverse events) for CDS.

Results We identified 102,443 people with diagnosed and 58,990 with undiagnosed HBP. We found that 21/35 (60%) of required concept sets were unused or inaccurate, with only 259 (25.3%) of 1,101 codes used. Use cases showed high inclusion (0.9–11.2%), low exclusion (0–0.1%), and missing patient-specific context (up to 65.6%), leading to data in 2/4 use cases being insufficient for accurate alerting.

Discussion Data quality from the EHR required to implement recommendations for HBP is highly inconsistent, reflecting a fragmented health care system and incomplete implementation of standard terminologies and workflows. Although imperfect, data were deemed adequate for two test use cases.

Conclusion Current data quality allows for further development of patient-facing FHIR HBP tools, but extensive validation and testing is required to assure precision and avoid unintended consequences.

Keywords

clinical decision support - hypertension - data quality

Background and Significance

High blood pressure (HBP) caused by essential hypertension is one of the most common conditions among adults in the United States.[1] HBP predicts major cardiovascular, renal, and cerebrovascular events.[2] Based on large-scale observational data, cardiovascular disease risk increases starting at 115/75 mm Hg, with a 12-point increase in average blood pressure (BP) doubling the risk of adverse cardiovascular outcomes.[3] The incidence of HBP has been increasing in the United States—46% of adults have stage 1 (130/80–139/89 mm Hg) or greater hypertension and 30% stage 2 (140/90–179/109 mm Hg) or worse hypertension. Simultaneously, our knowledge about lowering BP to avoid these outcomes has expanded.[1] Despite improving overall outcomes, BP control remains poor, with less than half of hypertensive adults meeting a goal <140/90 mm Hg.[1]

Clinical decision support (CDS) and care planning for complex conditions, including HBP, require substantial additional development steps to be optimized in clinical care. Poor data quality is a known issue with electronic health records (EHRs), both from the secondary use perspective and at the point of care.[4] [5] [6] Lack of conformance to standards, incompleteness of required or expected data elements, and implausible values are common in uncurated EHR data[7] and limit the usefulness, usability, and benefit of CDS tools. Berner et al evaluated EHR data quality for assessing risk of gastrointestinal bleeding; they found a completeness rate of 34% and a false negative rate for CDS firing of 77%.[8] Few other studies, however, have been conducted specifically to assess the impact of EHR data quality problems on the validity of CDS recommendations.

Without careful testing, CDS systems can have substantial unintended consequences. Patient-facing CDS may be especially susceptible to data quality problems, since patients are unlikely to have the resources or data access necessary to assess the face validity of data and CDS recommendations.[9]

The challenges in validating CDS for HBP are multifaceted, as HBP management is a highly variable process with complex information needs. Diagnosis and management of HBP requires accurate, serial measurements, including self-monitoring and use of specific protocols at home or the office to reduce the impact of inaccuracies.[10] Treatment of HBP includes both behavioral and lifestyle modifications (e.g., diet and exercise)—both infrequently recorded as structured data—as well as medications.[1] [11] [12] Furthermore, many patients with the highest cardiovascular risk also have high risk of adverse reactions to treatment or competing comorbidities, requiring tracking of these elements to address recommendations.[13] Adverse events, including falls, dizziness, metabolic derangements, and kidney dysfunction, come from observations, from clinical findings, and from laboratory measurements.[14] [15]

These complexities increase the data needed for CDS for HBP, and have limited electronic CDS implementations.[16] Although electronic HBP CDS systems have been created, few are in current use.[17] [18] [19] [20] [21] In addition, CDS trials for providers in HBP have been mixed. Hicks et al[20] showed no significant difference in HBP management using a CDS intervention, while several other CDS trials that involved multidisciplinary, multifaceted interventions—often including patient engagement and support—have had positive results.[17] [18] [21] [22]

In addition, HBP care planning is best when personalized to patient needs and experience, requiring significant data from and effort by patients.[23] [24] [25] [26] [27] Personalization is the goal of many digital health tools, but with mixed results.[28] [29] [30] [31] [32] For managing HBP, engaging patients for goal setting and shared decision-making is important yet requires accurate data about their conditions, treatments, and observations.[33] With the addition of Fast Healthcare Interoperable Resources (FHIR),[34] [35] delivering CDS to patients and care team is easier than ever before.[36] Guidelines can be transformed into FHIR implementations through clinical practice guidelines in FHIR (CPG-on-FHIR),[37] allowing for recommendations to be encoded for CDS with complex logic and patient-specific context. These changes are dependent on reliable, complete data to facilitate accurate and safe decision-making; with FHIR, however, understanding what data are available from an EHR is challenging.

Significance

To assess data quality issues, CDS must first have interoperable logic and standard value sets. Substantial work has attempted to standardize the process of building value sets and logic for CDS through the development of Clinical Quality Language to encode logic and value set repositories. Joint efforts from Agency for Research and Quality (AHRQ), National Institute of Diabetes and Digestive and Kidney Diseases, Centers for Disease Control and Prevention, Health Level Seven International, National Library of Medicine, and others have led to defined and vetted approaches such as CPG-on-FHIR and implementation guides for decision support for patients with several conditions, including chronic kidney disease (CKD), diabetes, cancer, and others.

Although these groups are providing standards, the issue of variable data specifications is still common.[38] Consistent data specifications are a historical barrier, as interoperable, valid ways to define sets of data elements for CDS require standardized coding for conditions (e.g., in International Classification of Diseases, 10th edition [ICD-10] or Systematized Nomenclature of Medicine [SNOMED] terminologies), for medications (in RxNorm), and for procedures (in SNOMED or Current Procedural Terminology [CPT]). Consistent use of these terminologies was uncommon until recently in the United States, further limiting CDS implementations across settings. Validation of interoperable data specifications may lead to quicker and more robust implementations of CDS, but standard validation is still a challenge for complex sets of recommendations. For HBP CDS, data required come from many different sources and must be personalized to patients' need.

Objective

In this study, our objective was to assess data quality and adequacy for specific recommendations for both patient and clinician management of HBP from multiple guidelines to prepare for a patient-facing FHIR application. We provide an explicit step-wise approach for others to follow by defining recommendations from guidelines and testing whether the data required for each step could be defined in a standard fashion and whether the data in an EHR were adequate to perform the logic for the recommendation.

Methods

We defined logic, value sets, and test use cases from HBP recommendations to understand whether data from an EHR would be sufficient to implement the CDS. We used standard approaches for value set creation, for bulk data extraction, and for data quality characterization. We then defined four use cases that exemplified the needed logic and explored how the CDS would run against the populations of interest over a year. For both, we used thresholds to determine likely problems with the data that would limit CDS accuracy.

Recommendations, Logic, and Value Set Creation

We used the work of Alper et al[39] to identify 71 recommendations from eight different hypertension guidelines. We then parsed these recommendations to identify key concepts required to assess the state of these recommendations on populations and patients. To assess the status of these concepts in the EHR, we used a five-pronged approach to identify previously used and/or validated data definitions available in EHR data and defined by FHIR. We used a combination of value sets from the Value Set Authority Center (VSAC; defined and used in clinical quality measures [CQMs]); from CDS Connect artifacts (used in other CDS); from phenotype definitions (used and validated to identify patient cohorts); from United Medical Language System services, including RxNAV for medications; and through the Observational Health Data Sciences and Informatics (OHDSI ATLAS) terminology services for missing or incomplete concepts. We chose the last based on the work of Hripcsak and coworkers for the LEGEND trial.[40] In this trial, they validated a set of encodings for key outcomes related to hypertension across a large dataset. The specific mappings and value sets are available from the open source GitHub repository for the project.[41]

Data Selection and Extraction

Data were derived from a 3.4 million patient EHR dataset from a large academic health center with three hospitals and more than 90 clinics. We limited initial patient look-up to those seen in ambulatory settings from 2010 to 2018 between age 18 and 85 (N = 2.1 million). We extracted data from the EHR by mimicking FHIR resource calls based on data domain. We did not use FHIR directly because we wanted the flexibility of searching for the data through nonstandard means. The resource domains included conditions that are mapped in both SNOMED with included hierarchical relationships (allowing children from the encoded SNOMED but not ancestors) and ICD codes; medications in RxNorm; observations and laboratory values in Logical Observation Identifiers Names and Codes (LOINC), and visits/utilization using SNOMED and CPT. We identified the initial populations in two ways. First, we found unique patients with the diagnosis of essential hypertension. Then, excluding those with a diagnosis, we found those with an elevated BP (>140/90) taken more than twice over more than two separate visits.

Data Quality Characterization

To assess the overall quality of data, we used Kahn et al's definitions of conformance and completeness.[7] Conformance is defined as the degree to which “data values adhere to specified standards and formats.” Completeness assesses whether the required or expected data values are present. Both data quality categories may be assessed using internal knowledge and information (verification) or external knowledge and information (validation).

We evaluated data conformance by first categorizing the value sets we found or created by their previous use and curation. We then mapped the internal data sources to these value sets and compared the use of these codes in the EHR data to the set of codes available. For completeness, we first looked at the prevalence of individual concepts across a population of those with hypertension, categorizing them as completely missing, extremely low (<0.1%), or low (<1%). Where structured EHR data were available but unmapped to standard concepts, we extracted these, flagged them as nonconforming, and still measured prevalence. Then, we examined the relative prevalence or incidence of key concepts for those with diagnosed hypertension and those with HBP (meeting diagnostic criteria) without a recorded diagnosis.

Data Adequacy for CDS Artifacts from Use Cases

For use cases, we used the recommendations to develop logical steps for the CDS, using frameworks for CQMs while adding specific requirements ([Table 1]). Similar to CQMs, CDS elements need information about the context and setting for the initial patient, then inclusion and exclusion criteria, the recommendation itself, and patient-specific context to provide more accurate decision-making. Selection of and consensus on the use-cases and value sets was done nominally through asynchronous review by authors.

Table 1 Constructs for determining validity for clinical quality measures and clinical decision support elements

Clinical quality measure construct	Clinical decision support construct
Initial patient population—on whom should you assess the denominator	Right context/setting where this CDS recommendation is valid
Denominator—# eligible for the measure	Included—# meeting requirements recommendation
Exclusions—# excluded from the denominator by default	Exclusions—# excluded by default
Numerator—# meeting the measure	Recommendation needed—# where recommendation is not met and should be shown
Exceptions—# who have a valid reason for not meeting the numerator (but are counted if they do meet it)	Patient-specific context—# who would receive this recommendation but may have other issues

Abbreviation: CDS, clinical decision support.

Instead of defining an initial patient population (as in CQMs), we defined the right context or setting to present the CDS. We used ambulatory visits with patients and providers for this criterion. For inclusion criteria, we used the recommendation inclusion (e.g., diagnosis of hypertension and BP not meeting goal). We separated criteria that may exclude patients from the recommendation to match the CQM categories. Next, we took the action or intervention implied by the recommendation and measured how many had received that intervention (akin to a numerator) and how many had not (where the recommendation would then be shown). Finally, we identified patient-specific context where the guideline identified potential variation or reasons that may influence—but not exclude—the decision to follow the recommendation. For instance, patients over 60 years may have different goals or recommendations based on preferences. We then created scripts to query the data for each stage of the use case, and used the percent of patients selected from the overall population as an indicator of the accuracy and feasibility of the use case. For instance, if a very small number or very large number of potential patients would meet criteria for a step, this was an indication that data may not adequately meet the use case. We then reviewed results as a group and qualitatively assessed issues related to data adequacy.

Results

We organized the 71 recommendations into five categories and seven data domains. The explicit presence of each data domain in the recommendation was counted ([Table 2]). Value sets or concepts from standard vocabularies were preferred, but nonstandard concepts were allowed. For the subcategories, demographics and conditions were required in 69 of 71; others ranged from 9 to 36.

Table 2 Recommendation categories and data domains and coding systems required

	Patient	Condition	Goal	Observation	Medication	Encounter/procedure	Laboratory test
Coding systems	CDCREC[a]	SNOMED[a]; ICD	LOINC[a]	SNOMED[a]; LOINC	RxNorm[a]	SNOMED[a]; CPT	LOINC[a]
Screening and testing (13 recs.)	13	13	1			3	11
Diagnosis (8 recs.)	6	6	1	7		6
Target (8 recs.)	8	8	8	8			2
Intervention—nonpharmacologic (6 recs.)	6	6		6
Intervention—pharm. (36 recs.)	36	36			36		2

Abbreviations: CPT, Current Procedural Terminology; HL7, Health Level 7; ICD, International Classification of Diseases; LOINC, Logical Observation Identifiers Names and Codes; RxNorm, a standardized nomenclature for clinical drugs; recs., recommendations; SNOMED, previously, the Systematized Nomenclature of Medicine.

^a Preferred coding system.

We defined 35 required concept sets (available in [Supplementary Material], available in the online version), 14 of which came from the National Library of Medicine's Value Set Authority Center, and 21 that we created from other sources. Sixteen of these 21 had prior validation (as described by the value set authority or documentation from their source), leaving 5 with no prior validation. The most common source of the other value sets was from the LEGEND study, which did predictive validation on their definitions through a set of observational cohort analyses.

Results from a global query of these concept sets in the EHR are shown in [Table 3]. The conformance column shows that five sets were unmapped. For instance, goals were available in the system, but not mapped to any standard codes.

Table 3 Data quality and adequacy for recommendations and outcomes by domain for essential hypertension cohort

Concept	Conformance mapping (# codes found in data/# codes in value set)	Completeness patients with 1+ code (N = 102,443)
Screening, testing, and diagnosis
BP per patient per year	LOINC (2/2 codes used)	3.2/pt/year
Home based blood pressure (ever)	SNOMED (0/1)	36 (0.01%)[a]
Ambulatory blood pressure (ever)	CPT (4/4)/SNOMED (1/4)	359 (0.35%)
Office BP by protocol	LOINC (0/1)[b]	0 (0%)
Goals and preferences	LOINC (0/3), SNOMED (0/5)[b]	708 (0.07%)[a]
BP target	LOINC (0/2)[b]	187 (0.02%)[a]
Interventions
Nonpharmacologic	SNOMED/CPT/ICD10 (34/107)	11,850 (11.6%)
Diet	SNOMED/CPT/HCPCS/ICD10 (4/17)	10,438 (10.2%)
Exercise	SNOMED/HCPCS/ICD10 (0/6)	0 (0%)
Weight loss	SNOMED/HCPCS (20/24)	1,224 (1.2%)
Smoking cessation	SNOMED/CPT (3/19)	1,080 (0.1%)
Alcohol	SNOMED (7/41)	230 (0.2%)
Pharmacologic
Any medication for HBP	RxNorm (62/103)[c]	86,792 (84.7%)
ACE-I	RxNorm (11/17)[b] [c] [d]	46,664 (45.6%)
ARB	RxNorm (13/13)[b]	23,529 (22.3%)
Thiazides	RxNorm (6/11)[b]	32,475 (31.7%)
CCB	RxNorm (15/28)[b]	36,625 (35.8%)
Other antihypertensive	RxNorm (17/34)[b]	32,660 (31.7%)
Outcomes
Hypertension treatment adverse events	SNOMEDCT/ICD/CPT (34/64)[b]	14,185 (13.9%)
Acute kidney injury	ICD (5/6)[b]	3,656 (3.6%)
Bradycardia	SNOMED/ICD (6/11)[b]	4,122 (4.0%)
Fall	ICD (6/9)[b]	2,692 (2.6%)
Hypotension	SNOMED/ICD (14/30)[b]	3,498 (3.4%)
Syncope	SNOMED (2/4)[b]	294 (0.3%)
Other	CPT (1/4)[b]	1,514 (1.5%)
Major adverse cardiovascular events (MACE)	SNOMED/ICD (122/805)[b]	20,691 (20.2%)
CHF	ICD (41/75)[b]	9,962 (9.7%)
ASCVD	ICD (46/156)[b]	4,587 (4.5%)
AMI	ICD (6/85)[b]	4,177 (4.1%)
Stroke	ICD (57/629)[b]	4,208 (4.1%)
CKD	ICD (18/260)[b]	19,424 (19.0%)

Abbreviations: AMI, acute myocardial infarction; ASCVD, atherosclerotic cardiovascular disease; BP, blood pressure; CCB, calcium channel blocker; CHF, congestive heart failure; CKD, chronic kidney disease; CPT, Current Procedural Terminology; ICD, International Classification of Diseases; LOINC, Logical Observation Identifiers Names and Codes; RxNorm, a standardized nomenclature for clinical drugs; SNOMED, previously, the Systematized Nomenclature of Medicine.

^a Codes not used but structured EHR data available through manual mapping.

^b Nonstandard value set.

^c Value set available but hierarchical RxClass more precise.

^d Taxonomy in EHR had error.

The other concept sets had related yet inadequate value sets from VSAC. For example, a value set of all antihypertensive medications was available but not at the class level; instead, we used RxClass to identify the class in the recommendations. Errors in internal EHR-loaded taxonomies were also found. For example, the angiotensin-converting-enzyme inhibitor Enalapril was not found in the extracted dataset; analysis found it linked as a specific ingredient of Enalaprilat. For conditions and procedures, SNOMED codes are preferred in standard taxonomies, yet we found 70% of total concepts in the ICD alone and 90 to 100% in CPT alone. For example, using SNOMED alone for CKD yielded 4,994 patients but using ICD-10 returned 19,416.

Interventions were heavily focused on medications; 84.7% of patients had pharmacologic interventions recorded as the prescription of one or more medication over the course of their illness. By contrast, only 11.6% of patients had nonpharmacologic interventions recorded. This indicates the lack of structured entry of these data; the precise adherence is unknown.

Data Validation by Cohort Identification and Prevalence

[Table 4] demonstrates two initial cohorts: the set with diagnosed hypertension (from [Table 3]), and a new set of patients with no diagnostic code and two or more elevated BPs over two or more visits (for screening recommendations). Patients were selected if they had a visit from 2010 to 2018 (N = 2.1 million); more recently seen patients (in 2018) were more likely to be in either cohort. Basic demographics, related conditions, and outcomes are given for both. The population demographics are similar by age and gender for broad hypertension prevalence, with an average age of 54 to 56.5. Racial and ethnic categories show an increased percentage of African-Americans diagnosed with hypertension; other categories were equivalent. Average BP in the undiagnosed group was higher. Nearly 60% of those without a recorded diagnosis received treatment for BP, indicating incomplete data for diagnosis. Estimating outcomes in patients with and without diagnosis was dramatically different; 20% of those with a hypertension diagnosis also had a recorded major adverse cardiovascular event, while only 9.1% of those with no diagnosis had a recorded event. Duration of contact was longer in those without a diagnosis and visits were less frequent, indicating less engagement with the health system.

Table 4 Cohort for analysis from 2.1 million patients seen from 2010 to 2018

	Patients with essential hypertension diagnosis	Patients with elevated BP but no diagnosis
N (% of 2.1 million patients) overall	102,446 (4.8%)	58,990 (2.8%)
N (% of 199,618) seen in 2018	37,513 (18.8%)	10,763 (5.4%)
% female sex	51,302 (50.1%)	27,463 (46.6%)
Average age	56.5 ± 18.8 y	54.3 ± 17.9 y
% Caucasian	88,872 (86.8%)	51,803 (87.8%)
% African American	3,456 (3.4%)	1,552 (2.6%)
% Asian	3,741 (3.7%)	1,201 (2.0%)
% Hispanic/Latino	6,061 (5.9%)	2,904 (6.1%)
Average SBP/DBP and SD	133.4 ± 14.9/75.9 ± 10.3	148.2 ± 9.2/81.8 ± 10.2
Related conditions
Secondary hypertension	2,550 (2.5%)	223 (0.4%)
Diabetes	24,757 (24.2%)	3,387 (5.7%)
Tobacco use	1,516 (1.5%)	216 (0.4%)
Pregnant	1,576 (1.5%)	392 (0.6%)
Pharm. treatment[a]: none	15,650 (15.3%)	22,983 (39.9%)
1 medication	19,855 (19.4%)	13,539 (23.0%)
2 medications	23,978 (23.4%)	11,302 (19.2%)
3 or more medications	42,960 (41.9%)	11,167 (18.9%)
Outcomes
Death	7,552 (7.4%); 1.3% mortality rate	2,954 (5.0%); 0.5% mortality rate
Major adverse cardiovascular  events	20,691 (20.2%)	5,390 (9.1%)
CHF	9,962 (9.7%)	948 (1.6%)
ASCVD	4,587 (4.5%)	1,503 (2.5%)
AMI	4,177 (4.1%)	719 (1.2%)
Stroke	4,208 (4.1%)	1,091 (1.9%)
CKD	19,424 (19.0%)	2,704 (4.6%)
Hospice	161 (0.2%)	14 (0.02%)
Duration of follow-up	5.6 ± 3.9 y	10.3 ± 6.8 y
Frequency of visits per year	1.9 ± 0.7 visits per year	0.3 ± 0.4 visits per year

Abbreviations: AMI, acute myocardial infarction; ASCVD, atherosclerotic cardiovascular disease; CHF, congestive heart failure; CKD, chronic kidney disease; DBP, diastolic blood pressure; SBP, systolic blood pressure; SD, standard deviation.

^a Over the course of their hypertension diagnosis; not necessarily concurrently.

Use cases and data adequacy: The four use cases defined involved diagnosis (1), goal setting, interventions for both nonpharmacologic (2) and pharmacologic (3) BP lowering, and special circumstances, including adverse events (4). The CDS components for each are provided in the figures, and red text indicates potential data quality issues that would impact CDS precision.

Use Case 1—Diagnosis via Home Blood Pressure Monitoring

Use case 1 generates substantial numbers of triggered patients, roughly 5% of patients seen in office visits. Few are excluded or have had coded versions of the recommended services. Patient context shows that many were already treated with antihypertensives, suggesting a missing diagnosis ([Fig. 1]).

Fig. 1 Data adequacy for hypertension diagnosis and out-of-office monitoring.

Use Case 2—Nonpharmacologic Recommendations

In use case 2, 9.5% of total patients seen would generate an alert as needing nonpharmacologic recommendations; this is largely due to the lack of coded goal setting and nonpharmacologic interventions ([Fig. 2]).

Fig. 2 Data adequacy for goal-setting.

Use Case 3—Pharmacologic Recommendations

Here, patients with a diagnosis over a goal of 130/80 may benefit from pharmacologic treatment according to several recommendations ([Fig. 3]). For this use case, a small number of patients would be identified (0.9%) and few had a history of adverse events. End-stage renal disease is lower than expected.

Fig. 3 Data adequacy for prescription of initial pharmacotherapy.

Use Case 4—Adverse Events and Goal Changes

When patients report adverse events, goals may be titrated upward to avoid further events. Patient-specific context may be a history of major adverse cardiovascular events, where the therapeutic index may be altered from hypertension alone ([Fig. 4]). Here, we see a small number would have the alert triggered (2.2%), and a large proportion have had an event in the past. Notably, 30 patients from this small group had goal changes already. Compared with all BP goals set (187/102,443, or 0.02%), providers are entering goals three times more frequently for those that may need the recommendation (30 per 199,618 patients seen yet 0.66% of 4,571).

Fig. 4 Data adequacy for adjustment of goals and treatment following adverse events.

Discussion

We found that it was possible to define 71 HBP recommendations and their required standardized data definitions. However, we had to develop or adjust 21 of 35 value sets for the data. Assessing the data quality in the EHR, we found that a substantial number of codes were infrequently or never used. For instance, goals were uncoded, limiting the ability to personalize CDS for individual patients. Similarly, some interventions related to exercise, smoking cessation, and alcohol use had limited mapping in the EHR, leading to two of four of the test use cases yielding high firing rates. Nonpharmacologic recommendations, for instance, would fire on 9.5% of patients; in a patient-facing application, this high rate may be appropriate, but care teams would likely experience significant fatigue.[42] Pharmacologic recommendations would fire on 0.9% of patients with HBP, indicating potential data adequacy. These mixed results show that implementation of CDS for HBP must have prior data quality and logic testing to avoid harm and alert fatigue.

These findings, while mixed, are improved from earlier CDS efforts, where every implementation had to be tailored to local data. Preferred terminologies (SNOMED, LOINC, and RxNorm) are now common in EHRs, even if data mappings are variable. Adapting based on data adequacy testing can improve CDS; for instance the majority of data were encoded to CPT and ICD rather than SNOMED, requiring developers to query both and perform extension mappings themselves. Patient-related concepts—goals, preferences, self-management interventions—had low standardization and use.

This work advances the literature in two ways: first, by implementing a standard method to test data adequacy or sufficiency for CDS. Second, by exploring some of the potential sources of alert fatigue. Based on our results, incomplete data would lead to much higher alerting rates in two use cases and would limit alerting to a small number in the other two use cases.[43] Implementation in either provider- or patient-facing CDS would have to account for the low rates of exclusions and patient-specific context/extenuating circumstances recorded in the data to mitigate the frequently reported alert fatigue from over-alerting.[44] [45] Specific data mapping by the local implementing site or substantial structured data collection would be required for some use cases; these processes are costly and time-intensive. Opportunities to gather the data directly from patients may reduce costs, as many of the missing elements are based on patient experience.[46] Others have shown positive feedback loops in managing HBP can be effective; Ralston et al showed secure messaging between patients and pharmacists or care managers improved BP[47]; while Benkert et al showed rapid feedback cycles improved BP control but highlighted the risk of message fatigue.[48]

There are several limitations to this work. First, the codes were solely pulled from a single EHR; other EHRs may have better mapping of concepts and differential recording based on workflow. For future work, we have produced the logical testing structure and relevant value sets for people to repeat our work with their own systems. We did not directly use FHIR, in part because bulk FHIR is not available locally, and using FHIR may generate different answers. We did reasonably conclude that many concepts would not be found using FHIR specifications because they are unmapped to the concepts; even mapped concepts may not be presented based on FHIR queries, so incidence would look even lower. In the future, we will produce open source FHIR specifications for these components. We did not use a formal process to define relevant value sets that we could not find. We encourage others to critique our choices and provide comments for improvement. Finally, we did not assess plausibility or accuracy of the data; the next step for our work is to perform chart reviews to develop these rules and assessments.

Implementers, innovators, and researchers are welcome to use our generated sets and test instructions as they build their own tools or check their own data, available at our GitHub site (https://github.com/mattStorer/OHSUHTNU18/tree/master/docs/resources/dataSufficiency). Future work will be to incorporate these lessons into CDS tools—both on the patient- and provider-facing side. The state of external applications to improve BP is still in flux, with applications that combine data and knowledge together in limited use.[49] Given current alerting rates would be extremely high for two of our use cases, future developers should understand many alerts are likely to be inaccurate, based on incomplete data, and a data completion effort must be a part of any future work. However, burnout of care teams limits further structured data entry,[50] [51] requiring more creative solutions.

Conclusion

Our work provides a framework to test data adequacy across value sets, between key populations, and across use cases. Gaps in data adequacy across these examples were common, and must be addressed prior to implementing CDS for HBP.

Clinical Relevance Statement

Data quality from the EHR required to provide care recommendations for HBP is highly inconsistent, with several use cases lacking adequate data quality for accurate alerting while other use cases have adequate data quality to proceed with testing and implementation.

Multiple Choice Questions

1. When defining data requirements for clinical decision support, what standard consideration did the authors use?

   • Using the smallest set of codes for any concept.

   • Previous validation of the set of codes chosen.

   • The shortest set of codes possible.

   • The use of the codes in other, non-EHR applications.

   Correct Answer: The correct answer is option b. Reusing previously validated value sets can improve accuracy, ease implementation burden, and help future evaluators by providing multisite estimates of likely precision. Code sets that vary from implementation to implementation for similar concepts both make implementation harder and can increase maintenance requirements.

2. In two of the use cases, large proportions of the initial patient population would receive alerts. What are possible impacts of this finding?

   • Users are likely to be grateful that their inaccurate coding is pointed out to them.

   • Increased numbers of alert always improve the overall quality of care and outcomes.

   • Increased frequency of alerts has been shown to increase alert fatigue, leading to missed high priority alerts and potential harm.

   • No obvious impacts from this finding, as it is irrelevant to clinical decision support performance.

   Correct Answer: The correct answer is option c. Alert fatigue is a common problem from clinical decision support, and inaccurate alerts that lead to higher frequency of alerts have been shown to reduce response to accurate and important alerts. These unintended consequences of clinical decision support were described more than a decade ago, and implementers must test their alerting systems for accuracy to minimize this issue.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank Brian Alper and Bryn Rhodes for reviewing earlier versions of the manuscript.

Protection of Human and Animal Subjects

Human and animal subjects were not included in this project. This work was approved by the Oregon Health and Science University Institutional Review Board.

• References

• 1 Fryar CD, Ostchega Y, Hales CM, Zhang G, Kruszon-Moran D. Hypertension prevalence and control among adults: United States, 2015-2016. NCHS Data Brief 2017; ; Oct (289) 1-8
• 2 Zonneveld TP, Richard E, Vergouwen MD. et al. Blood pressure-lowering treatment for preventing recurrent stroke, major vascular events, and dementia in patients with a history of stroke or transient ischaemic attack. Cochrane Database Syst Rev 2018; 7: CD007858
• 3 Lewington S, Clarke R, Qizilbash N, Peto R, Collins R. Prospective Studies Collaboration. Age-specific relevance of usual blood pressure to vascular mortality: a meta-analysis of individual data for one million adults in 61 prospective studies. Lancet 2002; 360 (9349): 1903-1913
• 4 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355
• 5 Thiru K, Hassey A, Sullivan F. Systematic review of scope and quality of electronic patient record data in primary care. BMJ 2003; 326 (7398): 1070
• 6 Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev 2010; 67 (05) 503-527
• 7 Kahn MG, Callahan TJ, Barnard J. et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC) 2016; 4 (01) 1244
• 8 Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annu Symp Proc 2005; 2005: 41-45
• 9 Glynn L, Casey M, Walsh J, Hayes PS, Harte RP, Heaney D. Patients' views and experiences of technology based self-management tools for the treatment of hypertension in the community: a qualitative study. BMC Fam Pract 2015; 16: 119
• 10 United States Preventive Services Task Force. Final recommendation statement: hypertension in adults: screening. April 27, 2021. Available at: https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/hypertension-in-adults-screening
• 11 Whelton PK, Einhorn PT, Muntner P. et al; National Heart, Lung, and Blood Institute Working Group on Research Needs to Improve Hypertension Treatment and Control in African Americans. Research needs to improve hypertension treatment and control in African Americans. Hypertension 2016; 68 (05) 1066-1072
• 12 Nguyen-Huynh MN, Hills NK, Sidney S, Klingman JG, Johnston SC. Race-ethnicity on blood pressure control after ischemic stroke: a prospective cohort study. J Am Soc Hypertens 2017; 11 (01) 38-44
• 13 Carey RM, Calhoun DA, Bakris GL. et al; American Heart Association Professional/Public Education and Publications Committee of the Council on Hypertension; Council on Cardiovascular and Stroke Nursing; Council on Clinical Cardiology; Council on Genomic and Precision Medicine; Council on Peripheral Vascular Disease; Council on Quality of Care and Outcomes Research; and Stroke Council. Resistant hypertension: detection, evaluation, and management: a scientific statement from the American Heart Association. Hypertension 2018; 72 (05) e53-e90
• 14 Qaseem A, Wilt TJ, Rich R, Humphrey LL, Frost J, Forciea MA. Clinical Guidelines Committee of the American College of Physicians and the Commission on Health of the Public and Science of the American Academy of Family Physicians. Pharmacologic treatment of hypertension in adults aged 60 years or older to higher versus lower blood pressure targets: a clinical practice guideline from the American College of Physicians and the American Academy of Family Physicians. Ann Intern Med 2017; 166 (06) 430-437
• 15 Wright Jr JT, Williamson JD, Whelton PK. et al; SPRINT Research Group. A randomized trial of intensive versus standard blood-pressure control. N Engl J Med 2015; 373 (22) 2103-2116
• 16 Prevention CfDCa. Interactive protocol for controlling hypertension. . Accessed July 5, 2021 at: https://nccd.cdc.gov/MillionHearts/Protocol/
• 17 Persson M, Mjörndal T, Carlberg B, Bohlin J, Lindholm LH. Evaluation of a computer-based decision support system for treatment of hypertension with drugs: retrospective, nonintervention testing of cost and guideline adherence. J Intern Med 2000; 247 (01) 87-93
• 18 Roumie CL, Elasy TA, Greevy R. et al. Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial. Ann Intern Med 2006; 145 (03) 165-175
• 19 Bosworth HB, Olsen MK, McCant F. et al. Hypertension Intervention Nurse Telemedicine Study (HINTS): testing a multifactorial tailored behavioral/educational and a medication management intervention for blood pressure control. Am Heart J 2007; 153 (06) 918-924
• 20 Hicks LS, Sequist TD, Ayanian JZ. et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med 2008; 23 (04) 429-441
• 21 Anchala R, Kaptoge S, Pant H, Di Angelantonio E, Franco OH, Prabhakaran D. Evaluation of effectiveness and cost-effectiveness of a clinical decision support system in managing hypertension in resource constrained primary health care settings: results from a cluster randomized trial. J Am Heart Assoc 2015; 4 (01) e001213
• 22 Rinfret S, Lussier MT, Peirce A. et al; LOYAL Study Investigators. The impact of a multidisciplinary information technology-supported program on blood pressure control in primary care. Circ Cardiovasc Qual Outcomes 2009; 2 (03) 170-177
• 23 Nair M, Ali MK, Ajay VS. et al. CARRS Surveillance study: design and methods to assess burdens from multiple perspectives. BMC Public Health 2012; 12: 701
• 24 Fix GM, Cohn ES, Solomon JL. et al. The role of comorbidities in patients' hypertension self-management. Chronic Illn 2014; 10 (02) 81-92
• 25 Fletcher BR, Hinton L, Hartmann-Boyce J, Roberts NW, Bobrovitz N, McManus RJ. Self-monitoring blood pressure in hypertension, patient and provider perspectives: a systematic review and thematic synthesis. Patient Educ Couns 2016; 99 (02) 210-219
• 26 Herrera PA, Moncada L, Defey D. Understanding non-adherence from the inside: hypertensive patients' motivations for adhering and not adhering. Qual Health Res 2017; 27 (07) 1023-1034
• 27 Ma C. An investigation of factors influencing self-care behaviors in young and middle-aged adults with hypertension based on a health belief model. Heart Lung 2018; 47 (02) 136-141
• 28 Omboni S, Ferrari R. The role of telemedicine in hypertension management: focus on blood pressure telemonitoring. Curr Hypertens Rep 2015; 17 (04) 535
• 29 Gibson B, Butler J, Schnock K, Bates D, Classen D. Design of a safety dashboard for patients. Patient Educ Couns 2020; 103 (04) 741-747
• 30 Goehringer JM, Bonhag MA, Jones LK. et al. Generation and implementation of a patient-centered and patient-facing genomic test report in the EHR. EGEMS (Wash DC) 2018; 6 (01) 14
• 31 Weissman GE, Yadav KN, Madden V. et al. Numeracy and understanding of quantitative aspects of predictive models: a pilot study. Appl Clin Inform 2018; 9 (03) 683-692
• 32 Ghimire S, Shrestha N, Callahan K. Barriers to dietary salt reduction among hypertensive patients. J Nepal Health Res Counc 2018; 16 (02) 124-130
• 33 Johnson RA, Huntley A, Hughes RA. et al. Interventions to support shared decision making for hypertension: a systematic review of controlled studies. Health Expect 2018; 21 (06) 1191-1207
• 34 International HLS. FHIR® – fast healthcare interoperability resources 2019,. November 1. Accessed July 5, 2021 at: http://hl7.org/fhir/
• 35 21st Century Cures Act, H.R. 34. 2016.
• 36 Services UDoHH. Pharmacist eCare Plan: HealthIT.gov. Accessed July 5, 2021 at: https://www.healthit.gov/techlab/ipg/node/4/submission/1376
• 37 Cinical Guidelines FHIR. (v0.1.0) (STU1 Ballot): HL7; 2019. Accessed July 5, 2021 at: http://hl7.org/fhir/uv/cpg/2019Sep/index.html#home
• 38 McClure RC, Macumber CL, Skapik JL, Smith AM. Igniting harmonized digital clinical quality measurement through terminology, CQL, and FHIR. Appl Clin Inform 2020; 11 (01) 23-33
• 39 Alper BS, Price A, van Zuuren EJ. et al. Consistency of recommendations for evaluation and management of hypertension. JAMA Netw Open 2019; 2 (11) e1915975
• 40 Suchard MA, Schuemie MJ, Krumholz HM. et al. Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis. Lancet 2019; 394 (10211): 1816-1826
• 41 Martijn Schuemie MS. OHDSI Large-Scale Evidence Generation and Evaluation in a Network of Databases (LEGEND) 2018. Accessed July 5, 2021 at: https://github.com/OHDSI/Legend
• 42 Aveyard P, Begh R, Parsons A, West R. Brief opportunistic smoking cessation interventions: a systematic review and meta-analysis to compare advice to quit and offer of assistance. Addiction 2012; 107 (06) 1066-1073
• 43 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 2007: 26-30
• 44 Gadhiya K, Zamora E, Saiyed SM, Friedlander D, Kaelber DC. Drug alert experience and salience during medical residency at two healthcare institutions. Appl Clin Inform 2021; 12 (02) 355-361
• 45 Olakotan OO, Mohd Yusof M. The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review. Health Informatics J 2021;27(02):14604582211007536
• 46 Perna G. Trend: patient-generated health data. Transforming data into decision support. The advance of patient-facing connected technologies will allow providers to track patients on a daily basis. Healthc Inform 2014; 31 (02) 26-27
• 47 Ralston JD, Cook AJ, Anderson ML. et al. Home blood pressure monitoring, secure electronic messaging and medication intensification for improving hypertension control: a mediation analysis. Appl Clin Inform 2014; 5 (01) 232-248
• 48 Benkert R, Dennehy P, White J, Hamilton A, Tanner C, Pohl JM. Diabetes and hypertension quality measurement in four safety-net sites: lessons learned after implementation of the same commercial electronic health record. Appl Clin Inform 2014; 5 (03) 757-772
• 49 Carrera A, Pifarré M, Vilaplana J. et al. BPcontrol. A mobile app to monitor hypertensive patients. Appl Clin Inform 2016; 7 (04) 1120-1134
• 50 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 51 Sieja A, Markley K, Pell J. et al. Optimization sprints: improving clinician satisfaction and teamwork by rapidly reducing electronic health record burden. Mayo Clin Proc 2019; 94 (05) 793-802

Publication History

Received: 15 January 2021

Accepted: 04 June 2021

Article published online:
04 August 2021