Is the Climb Worth the View? The Savings/Alert Ratio for Reducing Vitamin D Testing

Abstract

Background Despite guideline recommendations, vitamin D testing has increased substantially. Clinical decision support (CDS) presents an opportunity to reduce inappropriate laboratory testing.

Objectives and Methods To reduce inappropriate testing of vitamin D at the Vanderbilt University Medical Center, a CDS assigned providers to receive or not receive an electronic alert each time a 25-hydroxyvitamin D assay was ordered for an adult patient unless the order was associated with a diagnosis in the patient's chart for which vitamin D testing is recommended. The CDS ran for 80 days, collecting data on number of tests, provider information, and basic patient demographics.

Results During the 80 days, providers placed 12,368 orders for 25-hydroxyvitamin D. The intervention group ordered a vitamin D assay and received the alert for potentially inappropriate testing 2,181 times and completed the 25-hydroxyvitamin D order in 89.9% of encounters, while the control group ordered a vitamin D assay (without receiving an alert) 2,032 times and completed the order in 98.1% of encounters, for an absolute reduction of testing of 8% (p < 0.001).

Conclusion This CDS reduced vitamin D ordering by utilizing a soft-stop approach. At a charge of $179.00 per test and a cost to the laboratory of $4.20 per test, each display of the alert led to an average reduction of $14.70 in charges and of $0.34 in spending by the laboratory (the savings/alert ratio). By describing the effectiveness of an electronic alert in terms of the savings/alert ratio, the impact of this intervention can be better appreciated and compared with other interventions.

Keywords

clinical decision support alerting - order entry - quality improvement - clinical practice guidelines - ambulatory care

Background and Significance

Public interest in testing for and treating vitamin D deficiency has increased.[1] [2] While severe vitamin D deficiency affects bone health and has been studied in other disease processes (e.g., malignancy),[3] the benefit of screening the general population for vitamin D deficiency remains to be established.[4] Medical guidelines currently do not recommend screening average-risk individuals for vitamin D deficiency.[5]

Overuse of diagnostic laboratory testing contributes to the expense in modern medicine.[6] Health care systems increasingly utilize clinical decision support (CDS) to modify provider-ordering behavior and to assist in reducing the use of expensive treatment and diagnostic choices.[7] CDS has been shown to successfully lower inappropriate Clostridium difficile testing,[8] erythrocyte sedimentation rate assessments,[9] unnecessary imaging studies,[10] and duplicate laboratory testing.[11] CDS can reduce errors in order placement[12] and increase compliance with laboratory testing recommendations.[6]

Researchers lowered inappropriate vitamin D testing through reminding providers of vitamin D testing guidelines,[13] [14] by requiring providers to select an approved indication for testing to proceed,[15] [16] and by alerting providers if vitamin D testing had recently occurred.[17] We hypothesized that a sophisticated CDS that suppresses vitamin D testing alerts based on the diagnoses from the patient's chart would reduce inappropriate vitamin D testing.

Methods and Objectives

The Vanderbilt University Medical Center (VUMC) is an academic, tertiary care center in middle Tennessee, with clinics at the main campus and in several locations in and around Nashville, with approximately 2 million adults' ambulatory annual visit. A single–electronic health record system (Epic Systems, Verona, Wisconsin, United States) is used throughout the medical center.

To assess the efficacy of a clinical alert on inappropriate vitamin D testing, when ordering a 25-hydroxyvitamin D level, providers (physicians and midlevel providers) were assigned to receive (intervention) or not receive (control) an alert based on their unique VUMC provider-identification number. The decision support was triggered whenever a 25-hydroxyvitamin D assay was ordered in the outpatient setting. Upon entering the order, if the patient was at least 18 years of age and the provider was in the intervention group, the decision support compared the International Statistical Classification of Diseases and Related Health Problems (ICD-10) code(s) listed in the visit encounter to a list of conditions for which vitamin D testing is appropriate as per the Endocrine Society ([Table 1]).[5] If the ICD-10 code was matched to a diagnosis, no alert was triggered. If not, providers in the intervention group received a “pop up” alert suggesting that testing may not be appropriate. The alert was silent (not visible) for providers in the control group, which allowed for data collection for a comparison of how often providers in the control and intervention groups ordered 25-hydroxyvitamin D assays. [Fig. 1] contains a depiction of the CDS algorithm and the wording of the alert.

Table 1 Indications for vitamin D testing

Osteoporosis/osteopenia

Fracture (hip, wrist, or vertebral)

Osteomalacia/rickets/vitamin D deficiency

Chronic kidney disease/end-stage renal disease

Malabsorption (including gastric-bypass surgery, inflammatory bowel disease, celiac disease, and other disorders)

Hyperparathyroidism

Hypo- and hypercalcemia

Hypophosphatemia

Cirrhosis

Granulomatous disease (including sarcoidosis, lymphoma, tuberculosis, and other disorders)

Epilepsy

Human immunodeficiency virus infection

Obesity

Fig. 1 25-OHvitamin D = 25-hydroxyvitamin D. ICD-10, International Statistical Classification of Diseases and Related Health Problems.

During an 80-day period (October 24, 2018–January 12, 2019), collected data on all 25-hydroxyvitamin D assays ordered by providers included the presence or absence of a visible alert; completed orders; provider information such as role (attending, resident, midlevel provider, etc.), specialty (primary care or subspecialty), department, and location (clinic type); and basic patient demographic data. All 25-hydroxyvitamin D assays obtained throughout VUMC are processed at the central chemistry facility on the main campus. The direct, nonlabor expense to the laboratory for each 25-hydroxyvitamin D assay was $4.20 during the study period. The charge to the patient/payer was $179.00 per assay.

Encounters, during which a provider initiated the process of ordering a 25-hydroxyvitamin D assay, were the unit of analysis. Multiple order attempts during the same encounter (e.g., an order was started but then cancelled and then later reordered) were treated as a single event with the final outcome (order or no order during the encounter) recorded. The primary means of analysis was a Chi-square test utilizing 2 × 2 tables. The comparison of patients' ages utilized the Mann–Whitney U-test.

Results

Six hundred and sixteen outpatient providers initiated orders for a 25-hydroxyvitamin D assay during the 80 days, the decision support was active, with the total number of such orders per provider ranging from 1 to 154. Providers in the study group and the control group were similar, as were the patients whose providers were in the study or control groups ([Table 2]).

Table 2 Comparison of providers and patients by group

		Intervention group	Control group	p-Value
Providers[a]		322	294
	Attending physicians (%)	150 (46.6)	116 (39.5)	0.074
	Other providers (%)	172 (53.4)	178 (60.5)
	Primary care providers (%)	118 (36.8)	102 (34.7)	0.593
	Specialists (%)	203 (63.2)	192 (65.3)
Patients[a]		2181	2032
	White (%)	1597 (77.4)	1536 (79.1)	0.208
	Non-White (%)	466 (22.6)	407 (20.9)
	Male (%)	787 (36.1)	690 (34.0)	0.148
	Female (%)	1394 (63.9)	1342 (66.0)
	Mean age (y)	49.0	49.4	0.453

Note: % = percentage of group (intervention or control). Other providers = residents, fellows, midlevels providers, etc.

^a Not all providers had data on specialty available and not all patients had data on race available.

During the 80 days, providers initiated 12,368 orders for 25-hydroxyvitamin D assays. Of those, 4,213 (2,181 for the intervention group and 2,032 for the control group) adult patients were not linked to an appropriate diagnosis in the patient's chart and considered potentially inappropriate. Providers in the intervention group receiving the alert completed the vitamin D order 89.9% of the time, compared with 98.1% for providers in the control group, who did not receive the alert (p < 0.001; [Table 3]).

Table 3 Results

	Intervention group	Control group	p-Value
Patients	2,181	2,032
Order continued (%)	1,961 (89.9)	1,993 (98.1)	<0.001
Order discontinued (%)	220 (10.1)	39 (1.9)

Note: % = percent of effect on ordering (continued or discontinued).

In the intervention group, primary care providers who received the alert were more likely to discontinue vitamin D orders than subspecialists (order completed 88.0 vs. 90.9%, p = 0.031). In the intervention group, attending physicians discontinued the order equally as often as other providers such as resident physicians and midlevel providers (order completed 89.6 vs. 90.5%, p = 0.553; [Table 4]).

Table 4 Comparison of providers within the intervention subgroup

	Order continued	Order discontinued	p-Value
Patients[a]	1,961	220
Primary care providers (%)	692 (88.0)	94 (12.0)	0.031
Specialists (%)	1,264 (90.9)	126 (9.1)
Attending physicians (%)	1,296 (89.6)	150 (10.4)	0.533
Other providers (%)	665 (90.5)	70 (9.5)

Note: % = percent of effect on ordering (continued or discontinued). Other providers = residents, fellows, midlevels providers, etc.

^a Not all providers had data on specialty available.

For the 1,961 overrides of the alert, the providers in the intervention group selected “does not meet criteria” in 1,016 cases (51.8%) or “treating separate illness” in 750 cases (38.2%), with the additional selection of “see comments” in 193 cases (9.8%). Two overrides had no reasons recorded. For the 591 written comments provided (30.1% of overrides), the majority of the comments related to acceptable reasons for testing not well captured by visit diagnoses (e.g., ethnicities at increased risk for deficiency) or acceptable reasons that had not yet been added as a visit diagnosis.

Assuming the control order completion rate (98.1%), 2,140 orders would be expected in the intervention group, instead of the recorded 1,961. Thus, during the 80 days, the alert reduced vitamin D assay orders by 179 in the intervention group. At a cost of $4.20 per assay to the chemistry laboratory, this intervention reduced laboratory expenditure by $752. Extrapolating the results, expanding this intervention to all VUMC providers would lead to an estimated annual laboratory cost reduction of $6,620. Ignoring the cost of pain and inconvenience caused by blood draws to the patients and inconvenience and delays to the providers, based on a charge of $179 per assay to patients/payers (though actual payment is frequently less), the intervention reduced patient/payer spending by $32,000. Expanding this intervention to all providers would result in an estimated savings to patients/payers of $282,000 annually.

Discussion

Implementation of an alert reminding providers that a vitamin D assay may be inappropriate unless the patient had a relevant diagnosis led to a reduction in vitamin D testing, with more than 8% of intended tests never ordered. However, nearly 90% of the time the alert was displayed, the providers ordered the test anyway. Based on calculating the savings/alert ratio (savings in patient/payer spending divided by the number of alerts to achieve the savings), on average each alert saved $14.70. However, looking at the actual laboratory costs of running the assay, the savings were $0.34 per alert.

Testing vitamin D levels has increased worldwide,[18] with one study reporting a 94-fold increase in testing over a 4-year period in one location.[19] The costs associated with such testing are substantial,[18] with recent efforts made to develop a methodology for assessing the degree of inappropriate vitamin D testing.[20] In the United Kingdom, approximately 70% of vitamin D testing in one instance was inappropriate based on indication,[21] with another such study in Canada identifying a similar frequency of approximately 65%.[22] Vitamin D testing occurs despite a lack of guidelines supporting testing for general-risk individuals[4] [5] and the existence of scoring systems to identify high-risk individuals.[23]

Previous studies of CDS demonstrated a reduction in inappropriate vitamin D testing using a variety of strategies. Two studies reminded providers of conditions for which vitamin D testing is recommended at the time when a vitamin D level was ordered, reducing inappropriate testing from 43.8 to 30.3%[13] in one report and from 31 to 23% in another.[14] Several reports of an intervention in Canada that required providers to select a type of diagnosis from a list of conditions to proceed with testing both showed over a 90% reduction in the number of vitamin D tests ordered.[15] [16] An inpatient alert in one report increased appropriate retesting of vitamin D from 40 to 64% by alerting providers if the vitamin D level had been checked recently.[17] Our study used a different method, with providers, receiving an alert only if the patient did not have an appropriate diagnosis for the ordering of vitamin D testing.

The provider response to our intervention was weak. Providers typically overrode the alert and continued with ordering the vitamin D assays. This pattern commonly occurs, with a recent review reporting the “positive predictive value” of such alerts ranging from 8 to 83%.[24] We considered the following five potential explanations in our setting:

1. We worded the alert as an educational message, potentially reducing the force of the reminder.

2. We placed our alert early in the ordering process, at the time when the vitamin D order was placed. As our system allows providers to add diagnoses after ordering a laboratory, many providers may have supplied an appropriate diagnosis after the order was placed. Most providers selecting “treating separate illness” and some of the written comments may support this explanation. Additionally, relevant diagnoses from previous visits are not always maintained in the problem list, potentially contributing to visit diagnoses being added late in the work flow.

3. Our study did not utilize a hard stop, which has been shown to be effective in reducing inappropriate vitamin D testing.[13] [15] [16] [17] By simply alerting a provider that a test may be inappropriate, our alert served as a soft stop, likely accounting for the limited efficacy despite using a more specific trigger for displaying the alert than has been reported previously.

4. VUMC implemented its current electronic health record that started less than a year prior to this study. Not only did users have limited experience with this electronic health record, but multiple other electronic alerts remained in place from the time of its implementation, with outpatient providers receiving an average of 4.9 interruptive alerts per day.

5. Disagreement regarding appropriate vitamin D testing remains,[25] and VUMC providers may have disagreed with the provided indications for appropriateness.

In our analysis, comparing the number of times the alert was displayed and the overall financial savings, we created a savings/alert ratio that allows for a meaningful understanding of the financial impact of each display of an alert. While the actual cost saved to VUMC was negligible at $0.34 per alert, a displayed alert saved patients on average of $14.70. While $0.34 may be too low of a savings to warrant the interruption of a clinician by an alert, given the associated cognitive load and potential workflow disruptions,[26] a reduction of billed charges of $14.70 to patients for every alert displayed may warrant alerts. Of note, efforts to create and display the alert were shouldered by the organization, while patients were the beneficiary in a fee-for-service model. In an accountable, care model, VUMC and the patients' interests would have been better aligned. While our alert was not as effective as hoped based on the savings/alert ratio, we plan to improve savings by reformulating the message to be more directive and to reduce alerts by changing the firing of the alert to a point in the workflow where additional diagnoses are already recorded.

Strengths and Limitations

This study has several strengths worth noting. One is the large number of times the alert was displayed, allowing for confidence in the effect of the alert. Assigning providers and not patients to groups allowed us to not bias providers, who would have seen the alert in some patients and may have remembered it in others when not alerted. The control group allowed us to determine the effect of the alert. The diversity of providers included allowed for a comparison of the impact of an alert between primary care providers and specialists, as well as attending physicians, and other providers. By employing a soft-stop approach, provider autonomy was not eliminated in the care of patients.

This study has several additional limitations that must be detailed. By introducing an alert into a system already heavy with other alerts, this alert may have contributed to “alert fatigue.” By displaying the alert at the time the order was placed, the ordering provider may have been informed that an order might be inappropriate simply because the diagnosis had not yet been listed, potentially preventing appropriate vitamin D testing and also limiting our ability to determine the overall frequency of appropriate ordering of vitamin D. Adjusting the timing of this alert could address this problem.

Conclusion

In conclusion, we deployed a simple alert and demonstrated a measurable reduction in inappropriate 25-hydroxyvitamin D testing, though the timing of the alert likely limited its effectiveness. By quantifying the impact per alert (the savings/alert ratio), the financial impact of each display of an alert can be understood, and such a metric could be used as a point of comparison for the financial impact of disparate alerts and for decision making by governing bodies on retiring or maintaining alerts.

Clinical Relevance Statement

Inappropriate testing of vitamin D can be lowered through clinical decision support. To understand the impact of clinical decision support, the savings/alert ratio can be utilized.

Multiple Choice Questions

1. When designing interruptive alerts for clinicians, one way to compare the overall effectiveness may be to

   • Determine how often the alert is overridden.

   • Measure the length of time the alert is displayed.

   • Measure institutional alert fatigue.

   • Determine cost savings per alert.

   Correct Answer: The correct answer is option d. In this study, we were able to demonstrate the average savings per alert displayed providing a measure for its effectiveness.

2. Which of the following will NOT reduce ordering of unnecessary laboratory tests?

   • Removing the test from the orderable inventory catalogue.

   • Hard-stop alerts reminding the clinician that the test is not needed.

   • Soft-stop alerts reminding the clinician that the test is not needed.

   • Increasing the cost of the test.

   • Provider education.

   Correct Answer: The correct answer is option d. While option “a” is self-explanatory (if it doesn't exist, it cannot be ordered) both hard- and soft-stop alerts have been shown to reduce ordering of tests. Education will have a short-term effect on ordering of tests.

Conflict of Interest

C.U.L. reports Royalties for textbook publication from Springer Verlag. K.M.D. reports Consultancy fee, honoraria, clinical trial investigator from Alexion, Ultragenyx, and Moreo, grant from SoftBones, and personal fees for Clinical trial investigator from Regeneron.

Acknowledgments

The authors appreciate the input of Russell L. Rothman, MD, MPP, in the development of this project.

Protection of Human and Animal Subjects

The study was evaluated by VUMC's Institutional Review Board and granted an exemption, as it met criteria for a quality improvement initiative. This study was also reviewed and approved by VUMC's Laboratory Formulary Committee.

• References

• 1 Kotta S, Gadhvi D, Jakeways N. , et al. “Test me and treat me”--attitudes to vitamin D deficiency and supplementation: a qualitative study. BMJ Open 2015; 5 (07) e007401
• 2 Taylor CL, Thomas PR, Aloia JF, Millard PS, Rosen CJ. Questions about vitamin D for primary care practice: input from an NIH conference. Am J Med 2015; 128 (11) 1167-1170
• 3 Manson JE, Cook NR, Lee IM. , et al; VITAL Research Group. Vitamin D supplements and prevention of cancer and cardiovascular disease. N Engl J Med 2019; 380 (01) 33-44
• 4 LeFevre ML. ; U.S. Preventive Services Task Force. Screening for vitamin D deficiency in adults: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med 2015; 162 (02) 133-140
• 5 Holick MF, Binkley NC, Bischoff-Ferrari HA. , et al; Endocrine Society. Evaluation, treatment, and prevention of vitamin D deficiency: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab 2011; 96 (07) 1911-1930
• 6 Delvaux N, Van Thienen K, Heselmans A, de Velde SV, Ramaekers D, Aertgeerts B. The effects of computerized clinical decision support systems on laboratory test ordering: a systematic review. Arch Pathol Lab Med 2017; 141 (04) 585-595
• 7 Stenner SP, Chakravarthy R, Johnson KB. , et al. ePrescribing: reducing costs through in-class therapeutic interchange. Appl Clin Inform 2016; 7 (04) 1168-1181
• 8 White DR, Hamilton KW, Pegues DA, Hanish A, Umscheid CA. The impact of a computerized clinical decision support tool on inappropriate Clostridium difficile testing. Infect Control Hosp Epidemiol 2017; 38 (10) 1204-1208
• 9 Gottheil S, Khemani E, Copley K. , et al. Reducing inappropriate ESR testing with computerized clinical decision support. BMJ Qual Improv Rep 2016; 5 (01) u211376.w4582
• 10 Kanzaria HK, Hoffman JR, Probst MA, Caloyeras JP, Berry SH, Brook RH. Emergency physician perceptions of medically unnecessary advanced diagnostic imaging. Acad Emerg Med 2015; 22 (04) 390-398
• 11 Lippi G, Brambilla M, Bonelli P. , et al. Effectiveness of a computerized alert system based on re-testing intervals for limiting the inappropriateness of laboratory test requests. Clin Biochem 2015; 48 (16,17): 1174-1176
• 12 Yarahuan JW, Billet A, Hron JD. A quality improvement initiative to decrease platelet ordering errors and a proposed model for evaluating clinical decision support effectiveness. Appl Clin Inform 2019; 10 (03) 505-512
• 13 Felcher AH, Gold R, Mosen DM, Stoneburner AB. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc 2017; 24 (04) 776-780
• 14 Petrilli CM, Henderson J, Keedy JM. , et al. Reducing unnecessary vitamin D screening in an academic health system: what works and when. Am J Med 2018; 131 (12) 1444-1448
• 15 Ferrari R, Prosser C. Testing vitamin D levels and choosing wisely. JAMA Intern Med 2016; 176 (07) 1019-1020
• 16 Naugler C, Hemmelgarn B, Quan H. , et al. Implementation of an intervention to reduce population-based screening for vitamin D deficiency: a cross-sectional study. CMAJ Open 2017; 5 (01) E36-E39
• 17 Pelloso M, Basso D, Padoan A, Fogar P, Plebani M. Computer-based-limited and personalised education management maximise appropriateness of vitamin D, vitamin B12 and folate retesting. J Clin Pathol 2016; 69 (09) 777-783
• 18 Rockwell M, Kraak V, Hulver M, Epling J. Clinical management of low vitamin D: a scoping review of physicians' practices. Nutrients 2018; 10 (04) E493
• 19 Bilinski K, Boyages S. Evidence of overtesting for vitamin D in Australia: an analysis of 4.5 years of Medicare benefits schedule (MBS) data. BMJ Open 2013; 3 (06) e002955
• 20 Signorelli H, Straseski JA, Genzen JR, Walker BS, Jackson BR, Schmidt RL. Benchmarking to identify practice variation in test ordering: a potential tool for utilization management. Lab Med 2015; 46 (04) 356-364
• 21 Woodford HJ, Barrett S, Pattman S. Vitamin D: too much testing and treating?. Clin Med (Lond) 2018; 18 (03) 196-200
• 22 Rodd C, Sokoro A, Lix LM. , et al. Increased rates of 25-hydroxy vitamin D testing: Dissecting a modern epidemic. Clin Biochem 2018; 59: 56-61
• 23 Deschasaux M, Souberbielle JC, Andreeva VA. , et al. Quick and easy screening for vitamin D insufficiency in adults: a scoring system to be implemented in daily clinical practice. Medicine (Baltimore) 2016; 95 (07) e2783
• 24 Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of decision support in computerized provider order entry: systematic literature review. JMIR Med Inform 2018; 6 (01) e3
• 25 Pilz S, Zittermann A, Trummer C. , et al. Vitamin D testing and treatment: a narrative review of current evidence. Endocr Connect 2019; 8 (02) R27-R43
• 26 Sheehan B, Kaufman D, Bakken S, Currie LM. Cognitive analysis of decision support for antibiotic ordering in a neonatal intensive care unit. Appl Clin Inform 2012; 3 (01) 105-123