Appl Clin Inform 2021; 12(03): 429-435
DOI: 10.1055/s-0041-1730032
Case Report

Lessons Learned for Identifying and Annotating Permissions in Clinical Consent Forms

Elizabeth E. Umberfield
1  Health Policy & Management, Indiana University Richard M Fairbanks School of Public Health, Indianapolis, Indiana, United States
2  Center for Biomedical Informatics, Regenstrief Institute, Inc., Indianapolis, Indiana, United States
,
Yun Jiang
3  Department of Systems, Populations and Leadership, University of Michigan School of Nursing, Ann Arbor, Michigan, United States
,
Susan H. Fenton
4  School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, United States
,
Cooper Stansbury
5  Department of Computational Medicine and Bioinformatics, University of Michigan Medical School, Ann Arbor, Michigan, United States
6  The Michigan Institute for Computational Discovery and Engineering, University of Michigan, Ann Arbor, Michigan, United States
,
Kathleen Ford
3  Department of Systems, Populations and Leadership, University of Michigan School of Nursing, Ann Arbor, Michigan, United States
,
Kaycee Crist
7  Rory Meyers School of Nursing, New York University, New York, New York, United States
,
Sharon L. R. Kardia
8  Department of Epidemiology, University of Michigan School of Public Health, Ann Arbor, Michigan, United States
,
Andrea K. Thomer
9  University of Michigan School of Information, Ann Arbor, Michigan, United States
,
Marcelline R. Harris
3  Department of Systems, Populations and Leadership, University of Michigan School of Nursing, Ann Arbor, Michigan, United States
› Author Affiliations
Funding E. Umberfield was supported in part by the Robert Wood Johnson Foundation Future of Nursing Scholar's Program predoctoral training program. She is presently funded as a Postdoctoral Research Fellow in Public and Population Health Informatics at Fairbanks School of Public Health and Regenstrief Institute, supported by the National Library of Medicine of the National Institutes of Health under award number 5T15LM012502-04. The study was further supported by the National Human Genome Research Institute of the National Institutes of Health under award number 5U01HG009454-03, the Rackham Graduate Student Research Grant, and the University of Michigan Institute for Data Science. The content of this publication is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.

Abstract

Background The lack of machine-interpretable representations of consent permissions precludes development of tools that act upon permissions across information ecosystems, at scale.

Objectives To report the process, results, and lessons learned while annotating permissions in clinical consent forms.

Methods We conducted a retrospective analysis of clinical consent forms. We developed an annotation scheme following the MAMA (Model-Annotate-Model-Annotate) cycle and evaluated interannotator agreement (IAA) using observed agreement (A o), weighted kappa (κw ), and Krippendorff's α.

Results The final dataset included 6,399 sentences from 134 clinical consent forms. Complete agreement was achieved for 5,871 sentences, including 211 positively identified and 5,660 negatively identified as permission-sentences across all three annotators (A o = 0.944, Krippendorff's α = 0.599). These values reflect moderate to substantial IAA. Although permission-sentences contain a set of common words and structure, disagreements between annotators are largely explained by lexical variability and ambiguity in sentence meaning.

Conclusion Our findings point to the complexity of identifying permission-sentences within the clinical consent forms. We present our results in light of lessons learned, which may serve as a launching point for developing tools for automated permission extraction.

Protection of Human and Animal Subjects

Institutional Review Board review was not required because human subjects were not involved. Only blank consent forms were collected and analyzed.


Supplementary Material



Publication History

Received: 29 December 2020

Accepted: 31 March 2021

Publication Date:
23 June 2021 (online)

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany