Appl Clin Inform 2021; 12(01): 017-026
DOI: 10.1055/s-0040-1721010
Research Article

Leveraging Real-World Data for the Selection of Relevant Eligibility Criteria for the Implementation of Electronic Recruitment Support in Clinical Trials

Georg Melzer
1   Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
Tim Maiwald
2   Institute for Electronics Engineering, Department Electrical Engineering, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
Hans-Ulrich Prokosch
1   Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
Thomas Ganslandt
1   Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
3   Heinrich-Lanz-Center for Digital Health, Department of Biomedical Informatics, Mannheim University Medicine, Ruprecht-Karls-University Heidelberg, Mannheim, Germany
› Author Affiliations
Funding This study was funded in part by the European Commission within the EU/IMI project Electronic Health Records for Clinical Research (EHR4CR), grant no.: 115189.


Background Even though clinical trials are indispensable for medical research, they are frequently impaired by delayed or incomplete patient recruitment, resulting in cost overruns or aborted studies. Study protocols based on real-world data with precisely expressed eligibility criteria and realistic cohort estimations are crucial for successful study execution. The increasing availability of routine clinical data in electronic health records (EHRs) provides the opportunity to also support patient recruitment during the prescreening phase. While solutions for electronic recruitment support have been published, to our knowledge, no method for the prioritization of eligibility criteria in this context has been explored.

Methods In the context of the Electronic Health Records for Clinical Research (EHR4CR) project, we examined the eligibility criteria of the KATHERINE trial. Criteria were extracted from the study protocol, deduplicated, and decomposed. A paper chart review and data warehouse query were executed to retrieve clinical data for the resulting set of simplified criteria separately from both sources. Criteria were scored according to disease specificity, data availability, and discriminatory power based on their content and the clinical dataset.

Results The study protocol contained 35 eligibility criteria, which after simplification yielded 70 atomic criteria. For a cohort of 106 patients with breast cancer and neoadjuvant treatment, 47.9% of data elements were captured through paper chart review, with the data warehouse query yielding 26.9% of data elements. Score application resulted in a prioritized subset of 17 criteria, which yielded a sensitivity of 1.00 and specificity 0.57 on EHR data (paper charts, 1.00 and 0.80) compared with actual recruitment in the trial.

Conclusion It is possible to prioritize clinical trial eligibility criteria based on real-world data to optimize prescreening of patients on a selected subset of relevant and available criteria and reduce implementation efforts for recruitment support. The performance could be further improved by increasing EHR data coverage.


The present work was performed in fulfillment of the requirements for obtaining the degree “Dr. med.” from the Friedrich-Alexander University Erlangen-Nürnberg (FAU).

Protection of Human and Animal Subjects

The project was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the ethics board of the Medical Faculty of the University of Erlangen Nuremberg (247_14Bc).

Supplementary Material

Publication History

Received: 18 May 2020

Accepted: 04 October 2020

Article published online:
13 January 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Victor N. Registration of Clinical Studies from View of Ethics Committees (german language). Deutsches Ärzteblatt 2004; 101 (30) A-2111/B-1763/C-1695
  • 2 Cuggia M, Besana P, Glasspool D. Comparing semi-automatic systems for recruitment of patients to clinical trials. Int J Med Inform 2011; 80 (06) 371-388
  • 3 Schumacher M, Schulgen G. Controlled clinical trials - an introduction (german language). In: Methodology of clinical studies. 2008: 1-19
  • 4 Kalra D, Schmidt A, Potts H, Dupont D, Sundgren M, de Moor G. Case report from the EHR4CR project—A European Survey on Electronic Health Records Systems for Clinical Research. iHealth Connections 2011; 108-113
  • 5 Prescott RJ, Counsell CE, Gillespie WJ. et al. Factors that limit the quality, number and progress of randomised controlled trials. 1999; 3 (20) 1-143
  • 6 Fink T, Wicke D. Clinical trial - challenge with significant impact: (german language). Biotechnologie '10, '11 - Kapital, Markt, Wirtschaft. 2010. Accessed November 22, 2014
  • 7 EFPIA. The Pharmaceutical Industry in Figures: Key Data 2018. Accessed October 21, 2020 at:
  • 8 Getz KA, Stergiopoulos S, Short M. et al. The impact of protocol amendments on clinical trial performance and cost. Ther Innov Regul Sci 2016; 50 (04) 436-441
  • 9 Liu K, Acharya A, Alai S, Schleyer TK. Using electronic dental record data for research: a data-mapping study. J Dent Res 2013; 92 (7, suppl) 90S-96S
  • 10 Embi PJ, Jain A, Clark J, Harris CM. Development of an electronic health record-based clinical trial alert system to enhance recruitment at the point of care. AMIA Annu Symp Proc 2005; 2005: 231-235
  • 11 Köpcke F, Kraus S, Scholler A. et al. Secondary use of routinely collected patient data in a clinical trial: an evaluation of the effects on patient recruitment and data acquisition. Int J Med Inform 2013; 82 (03) 185-192
  • 12 De Moor G, Sundgren M, Kalra D. et al. Using electronic health records for clinical research: the case of the EHR4CR project. J Biomed Inform 2015; 53: 162-173
  • 13 Bruland P, Forster C, Breil B, Ständer S, Dugas M, Fritz F. Does single-source create an added value? Evaluating the impact of introducing x4T into the clinical routine on workflow modifications, data quality and cost-benefit. Int J Med Inform 2014; 83 (12) 915-928
  • 14 Dugas M, Lange M, Müller-Tidow C, Kirchhof P, Prokosch H-U. Routine data from hospital information systems can support patient recruitment for clinical studies. Clin Trials 2010; 7 (02) 183-189
  • 15 Bruland P, McGilchrist M, Zapletal E. et al. Common data elements for secondary use of electronic health record data for clinical trial execution and serious adverse event reporting. BMC Med Res Methodol 2016; 16 (01) 159
  • 16 Hersh WR, Weiner MG, Embi PJ. et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Med Care 2013; 51 (08) (Suppl. 03) S30-S37
  • 17 Weiner MG, Embi PJ. Toward reuse of clinical data for research and quality improvement: the end of the beginning?. Ann Intern Med 2009; 151 (05) 359-360
  • 18 Weng C, Tu SW, Sim I, Richesson R. Formal representation of eligibility criteria: a literature review. J Biomed Inform 2010; 43 (03) 451-467
  • 19 Blaisure J, Ceusters W. Business rules to improve secondary data use of electronic healthcare systems. Stud Health Technol Inform 2017; 235: 303-307
  • 20 Bache R, Taweel A, Miles S, Delaney BC. An eligibility criteria query language for heterogeneous data warehouses. Methods Inf Med 2015; 54 (01) 41-44
  • 21 Ash JS, Anderson NR, Tarczy-Hornoch P. People and organizational issues in research systems implementation. J Am Med Inform Assoc 2008; 15 (03) 283-289
  • 22 Ateya MB, Delaney BC, Speedie SM. The value of structured data elements from electronic health records for identifying subjects for primary care clinical trials. BMC Med Inform Decis Mak 2016; 16: 1
  • 23 Köpcke F, Trinczek B, Majeed RW. et al. Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence. BMC Med Inform Decis Mak 2013; 13: 37
  • 24 Doods J, Lafitte C, Ulliac-Sagnes N. et al. A European inventory of data elements for patient recruitment. Stud Health Technol Inform 2015; 210: 506-510
  • 25 Doods J, Botteri F, Dugas M, Fritz F. EHR4CR WP7. A European inventory of common electronic health record data elements for clinical trial feasibility. Trials 2014; 15: 18
  • 26 Löbe M, Stäubert S, Goldberg C, Haffner I, Winter A. Towards phenotyping of clinical trial eligibility criteria. Stud Health Technol Inform 2018; 248: 293-299
  • 27 Wang AY, Lancaster WJ, Wyatt MC, Rasmussen LV, Fort DG, Cimino JJ. Classifying clinical trial eligibility criteria to facilitate phased cohort identification using clinical data repositories. AMIA Annu Symp Proc 2018; 2017: 1754-1763
  • 28 Weng C. Optimizing clinical research participant selection with informatics. Trends Pharmacol Sci 2015; 36 (11) 706-709
  • 29 Rubin DL, Gennari J, Musen MA. Knowledge representation and tool support for critiquing clinical trial protocols. Proc AMIA Symp 2000; 724-728
  • 30 Ross J, Tu S, Carini S, Sim I. Analysis of eligibility criteria complexity in clinical trials. Summit On Translat Bioinforma 2010; 2010: 46-50
  • 31 Girardeau Y, Doods J, Zapletal E. et al. Leveraging the EHR4CR platform to support patient inclusion in academic studies: challenges and lessons learned. BMC Med Res Methodol 2017; 17 (01) 36
  • 32 Trinczek B, Köpcke F, Leusch T. et al. Design and multicentric implementation of a generic software architecture for patient recruitment systems re-using existing HIS tools and routine patient data. Appl Clin Inform 2014; 5 (01) 264-283
  • 33 Murphy SN, Weber G, Mendis M. et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010; 17 (02) 124-130
  • 34 Schreiweis B, Bergh B. Requirements for a patient recruitment system. Stud Health Technol Inform 2015; 210: 521-525
  • 35 Van Spall HGC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA 2007; 297 (11) 1233-1240
  • 36 Zhang H, He Z, He X. et al. Computable eligibility criteria through ontology-driven data access: a case study of hepatitis C virus trials. AMIA Annu Symp Proc 2018; 2018: 1601-1610
  • 37 Averitt AJ, Weng C, Ryan P, Perotte A. Translating evidence into practice: eligibility criteria fail to eliminate clinically significant differences between real-world and study populations. NPJ Digit Med 2020; 3: 67
  • 38 Ash N, Ogunyemi O, Zeng Q, Ohno-Machado L. Finding appropriate clinical trials: evaluating encoded eligibility criteria with incomplete data. Proc AMIA Symp 2001; 27-31
  • 39 Gehring S, Eulenfeld R. German medical informatics initiative: unlocking data for research and health care. Methods Inf Med 2018; 57 (S 01): e46-e49
  • 40 Baillie Gerritsen V, Palagi PM, Durinx C. Bioinformatics on a national scale: an example from Switzerland. Brief Bioinform 2019; 20 (Suppl. 02) 361-369
  • 41 Prokosch H-U, Acker T, Bernarding J. et al. MIRACUM: medical informatics in research and care in university medicine. Methods Inf Med 2018; 57 (S 01): e82-e91