Quantifying the Effect of Data Quality on the Validity of an eMeasure

Abstract

Objective The objective of this study was to demonstrate the utility of a healthcare data quality framework by using it to measure the impact of synthetic data quality issues on the validity of an eMeasure (CMS178—urinary catheter removal after surgery).

Methods Data quality issues were artificially created by systematically degrading the underlying quality of EHR data using two methods: independent and correlated degradation. A linear model that describes the change in the events included in the eMeasure quantifies the impact of each data quality issue.

Results Catheter duration had the most impact on the CMS178 eMeasure with every 1% reduction in data quality causing a 1.21% increase in the number of missing events. For birth date and admission type, every 1% reduction in data quality resulted in a 1% increase in missing events.

Conclusion This research demonstrated that the impact of data quality issues can be quantified using a generalized process and that the CMS178 eMeasure, as currently defined, may not measure how well an organization is meeting the intended best practice goal. Secondary use of EHR data is warranted only if the data are of sufficient quality. The assessment approach described in this study demonstrates how the impact of data quality issues on an eMeasure can be quantified and the approach can be generalized for other data analysis tasks. Healthcare organizations can prioritize data quality improvement efforts to focus on the areas that will have the most impact on validity and assess whether the values that are reported should be trusted.

Keywords

data quality - electronic health record - data quality assessment - ontology - quality

Protection of Human and Animal Subjects

De-identified EHR data were used for this research and proper precautions were taken to minimize privacy risk. Patients were allowed to opt out of having their medical data used for research. IRB approval was obtained (University of Minnesota IRB #1412E57982).

• References

• 1 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385
• 2 Zerhouni EA. Translational and clinical science--time for a new vision. N Engl J Med 2005; 353 (15) 1621-1623
• 3 Meystre SM, Lovis C, Bürkle T, Tognola G, Budrionis A, Lehmann CU. Clinical data reuse or secondary use: current status and potential future progress. Yearb Med Inform 2017; 26 (01) 1-15
• 4 Zozus MN, Hammond WE, Green BB. , et al. Assessing Data Quality for Healthcare Systems Data Used in Clinical Research [Internet]. NIH Collab. [cited March 1, 2016]. 2014:1–26. Available at: https://www.nihcollaboratory.org/Products/Assessing-data-quality_V1 0.pdf
• 5 Conway PH, Mostashari F, Clancy C. The future of quality measurement for improvement and accountability. JAMA 2013; 309 (21) 2215-2216
• 6 Torda P, Tinoco A. Achieving the promise of electronic health record-enabled quality measurement: a measure developer's perspective. EGEMS (Wash DC) 2013; 1 (02) 1031
• 7 Amster A, Jentzsch J, Pasupuleti H, Subramanian KG. Completeness, accuracy, and computability of National Quality Forum-specified eMeasures. J Am Med Inform Assoc 2015; 22 (02) 409-416
• 8 Agency for Healthcare Research and Quality. Measures Inventory [Internet]. 2015 [cited November 3, 2015]. Available at: http://www.qualitymeasures.ahrq.gov/hhs/matrix.aspx
• 9 Agency for Healthcare Research and Quality. Clinical Quality Measures [Internet]. 2015 [cited November 3, 2015]. Available at: https://ushik.ahrq.gov/QualityMeasuresListing?&system=mu&filterLetter=&resultsPerPage=50&filterPage=2&sortField=570&sortDirection=ascending&stage=Stage 2&filter590 = April 2014 EH&filter590 = July 2014 EP&enableAsynchronousLoading = true
• 10 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363: 501-504
• 11 Centers for Medicare & Medicaid Services (CMS). EHR Incentive Programs: 2015 through 2017 Overview [Internet]. 2015 [cited November 8, 2015]. Available at: https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Stage3Overview2015_2017.pdf
• 12 Persell SD, Wright JM, Thompson JA, Kmetik KS, Baker DW. Assessing the validity of national quality measures for coronary artery disease using an electronic health record. Arch Intern Med 2006; 166 (20) 2272-2277
• 13 HealthCatalyst. The Unintended Consequences of Electronic Clinical Quality Measures [Internet]. 2015 [cited November 8, 2015]. Available at: https://www.healthcatalyst.com/electronic-clinical-quality-measures-impact-data-quality
• 14 Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev 2010; 67 (05) 503-527
• 15 Centers for Medicare & Medicaid Services (CMS). Hospital Inpatient Quality Reporting (IQR) eCQM Validation Pilot Summary. 2016
• 16 Centers for Medicare & Medicaid Services (CMS). Clinical Quality Measures for CMS's 2014 EHR Incentive Program for Eligible Hospitals: Release Notes, April 1, 2013. 2014
• 17 Borsboom D, Mellenbergh GJ, van Heerden J. The concept of validity. Psychol Rev 2004; 111 (04) 1061-1071
• 18 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355
• 19 Kahn MG, Eliason BB, Bathurst J. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proc 2010; 2010: 356-360
• 20 Hasan S, Padman R. Analyzing the effect of data quality on the accuracy of clinical decision support systems: a computer simulation approach. AMIA Annu Symp Proc 2006; 324-328
• 21 Kahn MG, Brown JS, Chun AT. , et al. Transparent reporting of data quality in distributed data networks. EGEMS (Wash DC) 2015; 3 (01) 1052
• 22 Observational Medical Outcomes Partnership (OMOP) [Internet]. [cited July 15, 2015]. Available at: http://omop.org/
• 23 Overhage JM, Ryan PB, Reich CG, Hartzema AG, Stang PE. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc 2012; 19 (01) 54-60
• 24 Platt R, Carnahan RM, Brown JS. , et al. The U.S. Food and Drug Administration's Mini-Sentinel program: status and direction. Pharmacoepidemiol Drug Saf 2012; 21 (Suppl. 01) 1-8
• 25 Johnson SG, Speedie S, Simon G, Kumar V, Westra BL. A data quality ontology for the secondary use of EHR data. AMIA 2015 Annu Symp Proc 2015; 1937-1946
• 26 Staab S, Studer R. Handbook on Ontologies. Springer; 2010
• 27 Johnson SG, Speedie S, Simon G, Kumar V, Westra BL. Application of An Ontology for Characterizing Data Quality For a Secondary Use of EHR Data. Appl Clin Inform 2016; 7 (01) 69-88
• 28 CMS Clinical Quality eMeasure Logic and Implementation Guidance v1.3 [Internet]. 2014 [cited August 1, 2015]. Available at: https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/2014_eCQM_Measure_Logic_Guidancev13_April2013.pdf
• 29 Stéphan F, Sax H, Wachsmuth M, Hoffmeyer P, Clergue F, Pittet D. Reduction of urinary tract infection and antibiotic use after surgery: a controlled, prospective, before-after intervention study. Clin Infect Dis 2006; 42 (11) 1544-1551
• 30 Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46 (05) 830-836