Appl Clin Inform 2017; 08(01): 35-46
DOI: 10.4338/ACI-2016-09-CR-0148
Case Report
Schattauer GmbH

Using Active Learning to Identify Health Information Technology Related Patient Safety Events

Allan Fong
1   MedStar Institute for Innovation –National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, D.C. 20008, USA
,
Jessica L. Howe
1   MedStar Institute for Innovation –National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, D.C. 20008, USA
,
Katharine T. Adams
1   MedStar Institute for Innovation –National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, D.C. 20008, USA
,
Raj M. Ratwani
1   MedStar Institute for Innovation –National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, D.C. 20008, USA
2   Georgetown University Medical Center, 3800 Reservoir Rd NW, Washington, DC 20007
› Author Affiliations
FundingThis project was funded under contract/grant number Grant R01 HS023701–02 from the Agency for Health-care Research and Quality (AHRQ), U.S. Department of Health and Human Services. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the U.S. Department of Health and Human Services.
Further Information

Correspondence to:

Allan Fong
National Center for Human Factors in Healthcare
3007 Tilden St. NW, Suite 7M
Washington, D.C. 20008, USA
202–244–9807

Publication History

Received: 01 September 2016

Accepted: 09 January 2016

Publication Date:
20 December 2017 (online)

 

Summary

The widespread adoption of health information technology (HIT) has led to new patient safety hazards that are often difficult to identify. Patient safety event reports, which are self-reported descriptions of safety hazards, provide one view of potential HIT-related safety events. However, identifying HIT-related reports can be challenging as they are often categorized under other more predominate clinical categories. This challenge of identifying HIT-related reports is exacerbated by the increasing number and complexity of reports which pose challenges to human annotators that must manually review reports. In this paper, we apply active learning techniques to support classification of patient safety event reports as HIT-related. We evaluated different strategies and demonstrated a 30% increase in average precision of a confirmatory sampling strategy over a baseline no active learning approach after 10 learning iterations.


#

 


#

Conflict of interest

The authors have no conflict of interest.

  • References

  • 1 Charles D, Gabriel M, Searcy T, Carolina N, Carolina S. Adoption of Electronic Health Record Systems among U.. S. Non –Federal Acute Care Hospitals: 2008–2014. 2015
  • 2 Karsh BT, Weinger MB, Abbott PA, Wears RL. Health information technology: fallacies and sober realities. J Am Med Informatics Assoc 2010; 17 (Suppl. 06) 617-623. Available from: http://www.ncbi.nlm.nih.gov pubmed/20962121
  • 3 Fong A, Ratwani RM. An Evaluation of Patient Safety Event Report Categories Using Unsupervised Topic Modeling. Methods Inf Med 2015; 54 (Suppl. 04) 338-345.
  • 4 Magrabi F, Ong M-S, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Informatics Assoc 2012; 19 (Suppl. 01) 45-53. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21903979
  • 5 Borycki EM, Kushniruka W. Towards an integrative cognitive-socio-technical approach in health informatics: analyzing technology-induced error involving health information systems to improve patient safety. Open Med Inform J 2010; 4: 181-187.
  • 6 Walker JM, Hassol A, Bradshaw B, Rezaee ME. Health IT Hazard Manager Beta-Test.. Rockville, MD: 2012. Available from: https://healthit.ahrq.gov/sites/default/files/docs/citation/HealthITHazardManagerFinalReport.pdf
  • 7 Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. J Am Med Inform Assoc 2014; 21 e1 e28-e34. Available from: http://dx.doi.org/10.1136/amiajnl-2013–001762\nhttp://www.ncbi.nlm.nih.gov pubmed/24052536
  • 8 Walker JM, Hassol A, Bradshaw B, Rezaee ME. Health IT Hazard Manager Beta-Test.. Rockville, MD: 2012
  • 9 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Informatics Assoc 2013; 20 (Suppl. 05) 1-6. Available from: http://www.ncbi. nlm.nih.gov/pubmed/23666777
  • 10 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2015; (0) 1-7
  • 11 Amato MG, Salazar A, Hickman TT, Quist AJL, Volk LA, Wright A, McEvoy D, Galanter WL, Koppel R, Loudin B, Adelman J, McGreevey JD, Smith DH, Bates DW, Schiff GD. Computerized prescriber order entry –related patient safety reports: analysis of 2522 medication errors. J Am Med Informatics Assoc 2016; 0: 1-6.
  • 12 Settles B. Active Learning.. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool: 2012
  • 13 Settles B. Active Learning Literature Survey.. Madison (WI): University of Wisconsin-Madison. 2010 Jan 16. Computer Sciences Technical Report 1648.
  • 14 Kholghi M, Sitbon L, Zuccon G, Nguyen A. Active learning: a step towards automating medical concept extraction. J Am Med Informatics Assoc 2015; 0: 1-9. Available from: http://jamia.oxfordjournals.org/lookup/doi/10.1093/jamia/ocv069
  • 15 Zhang H, Huang M-L, Zhu X-Y. A Unified Active Learning Framework for Biomedical Relation Extraction. J Comput Sci Technol 2012; 27 (Suppl. 06) 1302-1313.
  • 16 Boström H, Dalianis H. De-identifying health records by means of active learning. In: Recall (micro).. 2012: 90-97.
  • 17 Clancy S, Bayer S, Kozierok R. Active Learning with a Human In The Loop.. Bedford, MA: 2012
  • 18 Settles B, Craven M, Friedland L. Active Learning with Real Annotation Costs.. Proceedings of the NIPS Workshop on Cost-Sensitive Learning 2008.
  • 19 Ong M-S, Magrabi F, Coiera E. Automated categorisation of clinical incident reports using statistical text classification. Qual Saf Health Care 2010; 19 (Suppl. 06) e55. Available from: http://www.ncbi.nlm.nih.gov pubmed/20724392
  • 20 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Inform Assoc 2013; 20 (Suppl. 05) 980-985. Available from: http://jamia.bmj.com/cgi/doi/10.1136/amiajnl-2012–001409
  • 21 Chang C, Lin C. LIBSVM: A Library for Support Vector Machines. ACM Trans Intell Syst Technol 2011; 2 (Suppl. 03) 27.
  • 22 Lewis DD, Catlett J. Heterogeneous Uncertainty Sampling for Supervised Learning.. In: Machine Learning: Proceedings of the Eleventh International Conference. 1994: 148-156.
  • 23 Manning CD, Raghavan P, Schütze H. Introduction to Information Retrieval.. Cambridge University Press: 2008
  • 24 Donmez P, Carbonell JG. Proactive Learning: Cost-Sensitive Active Learning with Multiple Imperfect Oracles.. In: Proceedings of the 17th ACM conference on Information and knowledge management. ACM 2008: 619-628.
  • 25 Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S, Xin D, Xin R, Franklin M, Zadeh R, Zaharia M, Talwalkar A. MLlib: Machine Learning in Apache Spark.. In: CoRR 2015.
  • 26 Kraska T, Talwalkar A, Duchi JC, Griffith R, Franklin MJ, Jordan MI. MLbase: A Distributed Machine-learning System.. In: CIDR 2013.

Correspondence to:

Allan Fong
National Center for Human Factors in Healthcare
3007 Tilden St. NW, Suite 7M
Washington, D.C. 20008, USA
202–244–9807

  • References

  • 1 Charles D, Gabriel M, Searcy T, Carolina N, Carolina S. Adoption of Electronic Health Record Systems among U.. S. Non –Federal Acute Care Hospitals: 2008–2014. 2015
  • 2 Karsh BT, Weinger MB, Abbott PA, Wears RL. Health information technology: fallacies and sober realities. J Am Med Informatics Assoc 2010; 17 (Suppl. 06) 617-623. Available from: http://www.ncbi.nlm.nih.gov pubmed/20962121
  • 3 Fong A, Ratwani RM. An Evaluation of Patient Safety Event Report Categories Using Unsupervised Topic Modeling. Methods Inf Med 2015; 54 (Suppl. 04) 338-345.
  • 4 Magrabi F, Ong M-S, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Informatics Assoc 2012; 19 (Suppl. 01) 45-53. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21903979
  • 5 Borycki EM, Kushniruka W. Towards an integrative cognitive-socio-technical approach in health informatics: analyzing technology-induced error involving health information systems to improve patient safety. Open Med Inform J 2010; 4: 181-187.
  • 6 Walker JM, Hassol A, Bradshaw B, Rezaee ME. Health IT Hazard Manager Beta-Test.. Rockville, MD: 2012. Available from: https://healthit.ahrq.gov/sites/default/files/docs/citation/HealthITHazardManagerFinalReport.pdf
  • 7 Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. J Am Med Inform Assoc 2014; 21 e1 e28-e34. Available from: http://dx.doi.org/10.1136/amiajnl-2013–001762\nhttp://www.ncbi.nlm.nih.gov pubmed/24052536
  • 8 Walker JM, Hassol A, Bradshaw B, Rezaee ME. Health IT Hazard Manager Beta-Test.. Rockville, MD: 2012
  • 9 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Informatics Assoc 2013; 20 (Suppl. 05) 1-6. Available from: http://www.ncbi. nlm.nih.gov/pubmed/23666777
  • 10 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2015; (0) 1-7
  • 11 Amato MG, Salazar A, Hickman TT, Quist AJL, Volk LA, Wright A, McEvoy D, Galanter WL, Koppel R, Loudin B, Adelman J, McGreevey JD, Smith DH, Bates DW, Schiff GD. Computerized prescriber order entry –related patient safety reports: analysis of 2522 medication errors. J Am Med Informatics Assoc 2016; 0: 1-6.
  • 12 Settles B. Active Learning.. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool: 2012
  • 13 Settles B. Active Learning Literature Survey.. Madison (WI): University of Wisconsin-Madison. 2010 Jan 16. Computer Sciences Technical Report 1648.
  • 14 Kholghi M, Sitbon L, Zuccon G, Nguyen A. Active learning: a step towards automating medical concept extraction. J Am Med Informatics Assoc 2015; 0: 1-9. Available from: http://jamia.oxfordjournals.org/lookup/doi/10.1093/jamia/ocv069
  • 15 Zhang H, Huang M-L, Zhu X-Y. A Unified Active Learning Framework for Biomedical Relation Extraction. J Comput Sci Technol 2012; 27 (Suppl. 06) 1302-1313.
  • 16 Boström H, Dalianis H. De-identifying health records by means of active learning. In: Recall (micro).. 2012: 90-97.
  • 17 Clancy S, Bayer S, Kozierok R. Active Learning with a Human In The Loop.. Bedford, MA: 2012
  • 18 Settles B, Craven M, Friedland L. Active Learning with Real Annotation Costs.. Proceedings of the NIPS Workshop on Cost-Sensitive Learning 2008.
  • 19 Ong M-S, Magrabi F, Coiera E. Automated categorisation of clinical incident reports using statistical text classification. Qual Saf Health Care 2010; 19 (Suppl. 06) e55. Available from: http://www.ncbi.nlm.nih.gov pubmed/20724392
  • 20 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Inform Assoc 2013; 20 (Suppl. 05) 980-985. Available from: http://jamia.bmj.com/cgi/doi/10.1136/amiajnl-2012–001409
  • 21 Chang C, Lin C. LIBSVM: A Library for Support Vector Machines. ACM Trans Intell Syst Technol 2011; 2 (Suppl. 03) 27.
  • 22 Lewis DD, Catlett J. Heterogeneous Uncertainty Sampling for Supervised Learning.. In: Machine Learning: Proceedings of the Eleventh International Conference. 1994: 148-156.
  • 23 Manning CD, Raghavan P, Schütze H. Introduction to Information Retrieval.. Cambridge University Press: 2008
  • 24 Donmez P, Carbonell JG. Proactive Learning: Cost-Sensitive Active Learning with Multiple Imperfect Oracles.. In: Proceedings of the 17th ACM conference on Information and knowledge management. ACM 2008: 619-628.
  • 25 Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S, Xin D, Xin R, Franklin M, Zadeh R, Zaharia M, Talwalkar A. MLlib: Machine Learning in Apache Spark.. In: CoRR 2015.
  • 26 Kraska T, Talwalkar A, Duchi JC, Griffith R, Franklin MJ, Jordan MI. MLbase: A Distributed Machine-learning System.. In: CIDR 2013.