Keywords electronic health records and systems - data warehousing and data marts - secondary
use - clinical trial - recruitment
Background and Significance
Background and Significance
Randomized clinical trials are a key component of medical research. Over 1,000,000
trials were performed since 1948.[1 ] They are regarded as gold standard to test new therapies and diagnostic techniques.[2 ]
[3 ]
(pp14–15) Many clinical trials cannot be conducted as planned though.[4 ]
[5 ] Slow recruitment and/or missed target cohort sizes often result in delays, cost
overruns, and even cancellation of clinical trials. With rising average costs of €0.8
billion in 2010[6 ] and €1.9 billion in 2016[7 ] for research, development and regulatory approval of a new active substance, each
failure can be a huge burden for the executing company or academic institution. Even
small amendments of the protocol can lead to costs of thousands of euros and delay
the trial by requiring ethics approvals for the amendment and implementing the changes
in the participating trial centers, as well as delayed time-to-market.[4 ]
[8 ] A good study design with a realistically estimated cohort size is essential to prevent
these issues. The increasing availability of structured patient data from electronic
health records (EHRs) provides new opportunities toward achieving these goals.
Benefits and Limitations of Real-World Data Use
The reuse of data acquired during routine care in EHRs has been shown to improve both
the correct estimation of cohort sizes, as well as the recruitment of study subjects.[9 ]
[10 ]
[11 ] The reported benefits include a simplified and better-targeted identification of
recruitment candidates,[10 ]
[11 ] higher rates of accrual,[12 ]
[13 ]
[14 ] as well as time savings, for the process of patient recruitment.[9 ]
[11 ]
[14 ] It has further been shown that secondary use of EHR data can prevent repeated data
reentry and improve data quality and cost-effectiveness of research.[13 ]
[15 ]
Several limitations regarding secondary use of routine data have been reported: documentation
for routine care and billing purposes may introduce selection biases, and their quality
and comprehensiveness may not be sufficient for research purposes.[16 ]
[17 ] Patients often visit several health providers, leading to fragmented EHRs.[17 ] The use of different terminologies (or the lack thereof) complicates and, in some
cases, precludes the merging of data from different sources both within an organization
or across institutional borders.[18 ]
[19 ]
[20 ]
[21 ]
Availability of Real-World Data
Several authors have examined how eligibility criteria from clinical trials overlap
with data items available in EHR systems. Ateya et al[22 ] decomposed eligibility criteria from 228 studies taken from a U.K. trial repository
and used expert classification to determine whether related EHR data elements could
likely be used; actual EHR data availability was not assessed. While they found that
74% of the criteria could likely be determined from EHR data, they also noted that
EHR queries on their own would be insufficient to determine recruitment and should
be seen as a tool to preselect patient cohorts for further manual screening. Köpcke
et al additionally assessed the actual availability of data items related to 15-investigator-initiated
trials in the EHRs of participating German hospitals and determined that on average,
only 35% of criteria were available, as well as documented.[23 ] Doods et al extracted inventories of commonly used eligibility criteria for feasibility
and recruitment from pharmaceutical trials in the Electronic Health Records for Clinical
Research (EHR4CR) project and examined the availability of corresponding data elements
at participating university hospitals.[24 ]
[25 ] While demographics, diagnosis, and procedure codes and a majority of laboratory
findings were highly available, most items from medical history, as well as scores
and classifications were rarely present. Löbe et al discussed the consequences of
this limited coverage:[26 ] patient cohorts based on a limited set of electronically available eligibility criteria
may overestimate the recruitable population by including false positives that need
to be eliminated by manual examination of (paper) patient charts.
Comprehensibility of Eligibility Criteria
Successful implementation of electronic recruitment support also depends on the quality
and computability of the eligibility criteria as they are defined in the study protocols.
Several publications have examined deficiencies of the current process of defining
eligibility criteria: clinical researchers often are not involved in clinical care
and documentation and may not have experience on whether certain items (e.g., “able
to swallow tablets” or “good health”) are routinely captured.[27 ] Also, a lack of precise understanding regarding etiologies and comorbidities and
their relevance to patient eligibility has been observed.[28 ] On the one hand, a focus on the principal (study) diagnosis may lead to overestimating
the size of the target cohort as possible exclusion criteria may not be taken into
account.[26 ] Researchers need to define cohorts of similar probands to ensure that study results
depend on the items of interest and no other confounding factors (confusion bias).[2 ] On the other hand, they need to exclude probands facing disproportionate risks by
participating in the study. Researchers often include prior experience in the definition
of eligibility criteria,[28 ] which can be subjective and unsystematic.[29 ] Post hoc, the detailed reasoning applied during selection of eligibility criteria
can often not be reconstructed which may complicate the detection and correction of
problems regarding patient enrollment.[28 ]
Ross et al examined a set of 1,000 trials randomly extracted from ClinicalTrials.gov
and assessed the comprehensibility (containing an interpretable criterion), selectiveness
(actually affecting candidate selection) and complexity (atomic versus combined criteria)
of their eligibility criteria.[30 ] They found 7% of the criteria to be incomprehensible or nonselective. Of the remaining
932 criteria, only 15% were found to be simple criteria containing discrete clinical
concepts in single phrases or quantitative comparisons. The other 85% were complex
criteria that contained multiple concepts, temporal constraints, or complex comparisons
that would require decomposition into distinct statements, as well as criteria that
would require clinical judgement, or information beyond the eligibility criteria (e.g.,
from the study protocol). Girardeau et al applied this classification to three studies
within the EHR4CR project and also assessed EHR availability and computability of
criteria.[31 ] They noted the influence of missing data on reducing the sensitivity (when regarding
inclusion criteria) and specificity (regarding exclusion criteria) of the queries.
Categorization of Eligibility Criteria
Wang et al implemented a similar approach toward the categorization of eligibility
criteria[27 ] by assessing the effort of electronic implementation:
“Easy” (supporting fully automated queries).
“Mixed” (supporting automated queries with subsequent manual checks).
“Hard” (requiring fully manual retrieval).
“Impossible” (not routinely documented in the EHR).
Using ClinicalTrials.gov, they found 292 individual criteria in a convenience sample
of 20 studies, removed duplicate and redundant criteria, and categorized them by six
independent observers from two separate research institutions, leading to the following
groups:
“Easy”: laboratory findings, diagnoses or procedures.
“Mixed”: diagnoses with modifiers (e.g., “severe cardiovascular disease [defined by
NYHA ≥ 3],” “active or untreated latent tuberculosis [TB]”).
“Hard”: criteria usually found in narrative clinical notes (e.g., “females who are
breastfeeding,” “eastern cooperative oncology group [ECOG] performance status of 0
or 1”).
“Impossible”: temporally related or generally undocumented criteria (e.g., “presenting
within timeframe for intravenous tPA treatment approved by local regulatory authorities
but no more than 4.5 hours from onset of symptoms,” “facial hair” or “good health”).
Role in Patient Recruitment
Trinczek et al proposed a generic software architecture for patient recruitment systems
(PRS)[32 ] consisting of five modules: trial administration module, notification module, patient
data module, query module, and screening list module. In the query module, eligibility
criteria transferred from the trial administration module are converted to executable
queries, which are then applied to the patient data module. The authors also posited
that the selection of the eligibility criteria to implement electronically is crucial.
We have added this activity as a separate step in the recruitment architecture diagram
proposed by Trinczek et al between the trial administration and query modules ([Fig. 1 ]).
Fig. 1 Generic patient recruitment support software architecture (modified after Trinczek
et al[32 ]), supplemented with a criteria selection step.
Cuggia et al have also emphasized the critical nature of clearly formulated and interpretable
eligibility criteria.[2 ] They have posited that prescreening is a crucial step of the recruitment process.
Prescreening is described as an initial selection of potential candidates from a proband
population based on a subset of prioritized eligibility criteria. Only subsequently,
candidates are screened against the full set of criteria. Even though Cuggia et al
reviewed 28 recruitment support publications, none of the eight, which were presented
in detail, described a process of how the prioritized subset was selected. In this
paper, we propose a stepwise approach toward the selection of such a prioritized subset.
Objectives
In this project, we analyzed the eligibility criteria of a clinical trial with the
goal of developing a systematic approach toward identifying a relevant subset of criteria
best suited to implement recruitment support, based on availability in the EHR and
their discriminatory power. To our knowledge, no systematic approach toward selection
of relevant eligibility criteria for recruitment support has been published so far.
The project was performed within the EHR4CR project, a European Union Innovative Medicines
Initiative (EU-IMI) funded public–private partnership, which focused on the optimization
of clinical trials throughout the feasibility, recruitment, execution, and pharmacovigilance
phases.[12 ]
Methods
Based on trials by the participating pharma companies that were actively recruiting
during the project phase, the KATHERINE study was selected for the project (NCT01772472).
The KATHERINE study compares the efficacy and safety of trastuzumab–emtansine versus
trastuzumab in breast cancer for patients with HER2-positive residual tumors after
tumor resection and neoadjuvant therapy.
Permission to carry out the study was granted by the ethics board of the Medical Faculty
of the Friedrich-Alexander University Erlangen-Nürnberg (247_14Bc). All eligibility
criteria were extracted from the study protocol and not from the simplified version
available at ClinicalTrials.gov. They were classified according to the comprehensibility,
selectiveness, and complexity aspects by Ross et al[30 ] and compared with the previously published EHR4CR data inventories.[15 ]
[24 ]
[25 ] The eligibility criteria were then simplified to allow algorithmic implementation.
To prepare the eligibility criteria for electronic execution, we classified and refined
them according to Ross et al[30 ] in the following stepwise approach:
Identification of incomprehensible or nonselective criteria which were eliminated
from further processing.
Identification of duplicate criteria (also covering inclusion criteria which were
duplicated as inversely formulated exclusion criteria) which were reduced to a single
instance.
Identification of noncomputable criteria (e.g., requiring physician interpretation)
which were eliminated from further processing.
Identification of complex criteria (i.e., containing several attributes in a single
clause) which were decomposed into simple criteria.
Based on the respective sections of the study protocol, criteria were also tagged
as disease specific versus nondisease-specific.
Eligibility criteria were then matched with data elements from the local clinical
data warehouse at Erlangen University Hospital, a tertiary-care academic site with
1,394 beds. While the hospital offers specialized outpatient clinics, ambulatory care
in Germany is covered primarily through general practitioners not affiliated with
the hospitals. Data warehouse content thus relates mostly to inpatient care. A preselection
of relevant patient identifiers was extracted from the local tumor documentation system
(GTDS, Gießen University) based on documented breast cancer (ICD code “C50”) and neoadjuvant
therapy (T-stage “y” in the TNM classification of malignant tumors) during the time
period from March 25, 2013–October 27, 2014. Available data elements for the selected
cohort were exported from an i2b2 platform.[33 ] At the same time, a paper chart review was performed for the same cohort to manually
extract all documented eligibility data elements.
The availability of all eligibility data elements was calculated both for the data
warehouse extract, as well as the data manually extracted by chart review. Data element
availability was compared with the EHR4CR feasibility data inventory.[25 ]
The refined set of eligibility criteria for electronic execution consisted only of
comprehensible, selective and simple criteria. Data availability was computed for
all criteria in the set based both on the dataset generated from chart review, as
well as from the data warehouse. To quantify discriminatory power of criteria, an
isolated inclusion or exclusion result was determined for each available value in
both datasets. Missing values were considered to be “neutral” in the sense of resulting
neither in an inclusion or exclusion. A comparison was performed between the eligibility
results of both data sources. In case of discrepancies, the reasons were determined
and documented based on reviewing the patient chart and raw data in the clinical data
warehouse. Also, the specificity of the combined disease-specific and nondisease-specific
criteria were calculated, respectively. In a further step, we applied a score to select
criteria most suitable for electronic execution, based on the following components:
Disease specificity: criteria listed in the “disease-specific eligibility” sections
of the protocol received a point.
Data availability: criteria which had data available from the paper chart or data
warehouse received a point.
Discriminatory power: criteria which were discriminatory (i.e., with available data
leading to patient exclusion) received a point.
The score was added for each criterion, and a threshold of 2 was defined for inclusion
into the final set of eligibility criteria for electronic execution.
The screening list was obtained from the principal investigator to provide the patients
actually included in the study. The inclusion list was used to calculate sensitivity
and specificity for the selected eligibility criteria. Classification and scoring
of eligibility criteria, as well as the paper chart reviews, were performed by a fifth-year
medical student (G.M.) and vetted by a medical doctor (T.G.).
Results
The selection and refinement of eligibility criteria is shown in [Fig. 2A, B ]. All 35 original eligibility criteria were determined to be comprehensible, with
1.5 criteria being nonselective and 33.5 selective according to the taxonomy described
by Ross et al[30 ] (“fractional” criteria are given when a complex criterion contains multiple simple
criteria with different classifications). Of these, 10 criteria were classified as
“simple” and 23.5 as “complex.” After the removal of duplicates (3) and noncomputables
(7) and the decomposition of the complex criteria into simple components, a total
of 70 individual criteria resulted ([Table 1 ]; [Supplementary Table S1 ] [available in the online version]; for the detailed list). Data elements for 53
criteria (75.7%) were available in the local clinical data warehouse. 47 (67.1%) items
were included in the EHR4CR trial feasibility inventory,[25 ] 48 items (68.6%) were part of the EHR4CR recruitment inventory,[24 ] and 47 items (67,1%) were part of the EHR4CR trial execution inventory.[15 ]
Fig. 2 (A ) Flowsheet describing the selection and refinement of eligibility criteria, starting
from the original criteria as given in the study protocol. After a categorization
step, non-selective, duplicate and non-computable criteria are removed. Simplification
of the remaining criteria leads to a set of 70 criteria, of which 17 are selected
after scoring (see [Fig. 2B ] for details). (B ) Flowsheet describing the scoring and selecting criteria to achieve a prioritized
subset. Starting from a cohort of breast cancer patients with neoadjuvant first-line
therapy, data for the full set of simplified eligibility criteria is extracted in
parallel from a data warehouse query and a paper chart review. The eligibility impact
of each available data element is annotated to derive a master data table giving the
availability and eligibility impact of each data element for all patients from both
sources. A score is calculated based on the disease-specificity, data availability
and discriminatory power of each criterion.
Table 1
Composition of eligibility criteria; “fractional” criteria are given when a complex
criterion contains several simple criteria with different classifications
Original set
35 criteria n (%)
•Nonselective criteria
1.5 (4.3)
•Selective criteria
33.5 (95.7)
▪ Simple
10 (28.6)
▪ Complex
23.5 (67.1)
Categorization of selective criteria
33.5
•Duplicate criteria
3 (9.0)
•Noncomputable criteria
7 (20.9)
•Nonduplicate, computable criteria
23.5 (70.1)
Decomposed (simplified) set
70 criteria
•Available in local data warehouse
53 (75.7)
•Present in EHR4CR feasibility criteria inventory[25 ]
47 (67.1)
•Present in EHR4CR recruitment criteria inventory[24 ]
48 (68.6)
•Present in EHR4CR trial execution inventory[15 ]
47 (67.1)
Abbreviation: EHR4CR, Electronic Health Records for Clinical Research.
The preselection of relevant patients from the GTDS tumor documentation system yielded
115 patient identifiers. Of these, 106 (92%) paper charts could be retrieved during
the study period, whereas 9 (8%) charts were unavailable due to clinical use. These
patients were excluded from the project. Manual chart review to extract the computable
data items took 32.5 hours (18.8 minutes on average). It took 15.5 hours to determine
availability of data items and extract them from the clinical data warehouse. Out
of a total of 7,420 possible data elements (70 items for 106 patients), 3,551 (47.9%)
were available from the paper charts and 1,995 (26.9%) from the data warehouse.
[Fig. 3 ] shows the data availability for the paper chart review and data warehouse, aggregated
by the groups from the EHR4CR recruitment inventory and compared with the availability
listed there[24 ] ([Supplementary Figs. S1 ] and [S2 ] [available in the online version] for a detailed breakdown). [Table 2 ] shows data availability for these groups in the local data warehouse compared with
the EHR4CR feasibility and trial execution inventories. Eligibility results were determined
for each individual data item for both sources ([Fig. 4 ]). Results were concordant for 1,930 data elements (99.7%) and differed in five cases
(0.3%). The reasons for these differences were determined and are given in [Table 3 ].
Fig. 3 Data availability for the paper chart review and data warehouse, aggregated by the
groups from the EHR4CR recruitment inventory and compared with the availability listed
there[24 ] ([Table 2 ] for accompanying data). EHR4CR, Electronic Health Records for Clinical Research.
Table 2
Accompanying data table for [Fig. 3 ] showing data availability from paper chart and data warehouse and the Inventory
by Doods et al[25 ]; availability from the inventory was averaged across the nine participating hospitals
Criteria group
Paper chart review (%)
Data warehouse query (%)
Inventory by Doods et al (%)
Demographics
100.0
100.0
88.6
Medical history
30.2
0.5
18.9
Diagnosis
0.8
0.1
61.0
Procedure
88.2
88.2
79.6
Findings
68.1
0.0
20.2
Laboratory findings
71.9
59.7
81.8
Medication
35.5
9.7
60.0
Scores or classification
69.4
27.8
0.0
Fig. 4 Isolated eligibility results of individual eligibility criteria grouped by inventory
by Doods et al,[24 ] paper chart review and the data warehouse query.
Table 3
Mismatching criteria between paper charts and data warehouse
Criterion
Affected patients
Comment
TNM (post-op)—T/TNM (post-op)—R
3
The data warehouse query returned data for earlier pathology findings which fell into
the relevant time window
laboratory value—hemoglobin
1
The data warehouse query returned an earlier finding
anticancer drug—epirubicin, cycles
1
The paper charts contained more detailed information including the epirubicin cycles
(leading to exclusion), whereas the data warehouse contained only the general information,
that epirubicin was given
Specificity of the combined disease-specific criteria was 0.55 and 0.13 for the combined
nondisease-specific criteria. Application of the scoring system led to 2 criteria
receiving the maximum score of 3 points, 15 criteria with 2 points, 37 criteria with
1, and 16 criteria with 0 points. Based on the cut-off at 2 points, a set of 17 criteria
were selected for electronic execution ([Table 4 ]).
Table 4
Number of criteria in relation to the score as well as sensitivity and specificity
calculated for each combination both from the data warehouse query and the paper chart
review
Number of criteria
Data warehouse query
Paper chart review
Sens.
Spec.
Sens.
Spec.
Score 3
2
1.000
0.554
1.000
0.772
Score 2 or above
17
1.000
0.574
1.000
0.802
Score 1 or above
54
1.000
0.574
1.000
0.811
Application of the criteria against the data warehouse dataset and screening list
yielded a sensitivity of 1.00, a specificity of 0.57, a positive predictive value
of 0.10, and a negative predictive value of 1.00 ([Table 5 ]).
Table 5
Contingency table of inclusion and exclusion of patients comparing electronic execution
(PRS recommendation) of eligibility criteria from the data warehouse versus the trial
screening list
Trial screening list
Inclusion
Exclusion
PRS recommendation
Inclusion
5
43
Exclusion
0
58
Note: Sensitivity: 1.00, specificity: 0.57, positive predictive value: 0.10, negative
predictive value: 1.00.
Discussion
Prescreening has been described as an essential but challenging step within the recruitment
process, facilitating an initial selection of potentially recruitable patients from
a base population,[2 ]
[32 ]
[34 ] based on a limited set of criteria available from electronic sources and followed
up by in-depth manual review of the candidates against the full set of eligibility
criteria. This aligns with the expectations of potential users of electronic recruitment
support in the sense that a PRS would not be expected to provide a definite list of
patients to be recruited, but rather a relevant preselection for further manual inspection.[32 ]
Applying the categorization proposed by Ross et al[30 ] to KATHERINE study, the composition of eligibility criteria was similar to that
reported by Ross et al: 95.7% of criteria were comprehensible (Ross et al: 93.2%),
and among those 72% were complex and 28% were simple criteria (Ross et al: 85/15%).
We applied a stepwise process of categorizing, pruning and electronic implementation
of criteria that allowed us to reduce the effort required for setting up electronic
recruitment support. We support the recommendation by Ross et al and van Spall et
al[30 ]
[35 ] that eligibility criteria should be formulated in a comprehensible, selective, and
simple manner to provide a concise set of consistent criteria suitable for electronic
implementation. In a more recent project, Zhang et al analyzed eligibility criteria
from 77 Hepatitis C-Virus (HCV) trials in 2018,[36 ] finding 85% of criteria to be computable and proposed a classification of eligibility
criteria related to their ontology-based operationalization. Combining simplification
based on Ross et al and prioritization as described in this paper with an ontology-based
implementation could improve generalizability, for example, regarding application
across different terminologies and granularities of clinical data.
Even though previous publications have stated the importance of prioritizing relevant
eligibility criteria for the prescreening step,[2 ] according to our knowledge, no concrete process has been published regarding how
to select this relevant subset. We additionally applied a scoring system to select
a subset of criteria most relevant for building candidate lists for electronic recruitment
support and evaluated it based on a comparison with the patients actually recruited
into the trial. We performed a preselection of relevant patients based on core eligibility
criteria (breast cancer and neoadjuvant therapy). This step provided us with a dataset
that we could then analyze in relation to the availability and value distribution
of the remaining eligibility criteria specific to that selected cohort. We chose to
prioritize disease-specific criteria as their combined specificity was higher (0.55)
than the combined nondisease-specific criteria (0.13). Additionally, we prioritized
criteria for which the available data in the preselected cohort was nonuniform regarding
inclusion or exclusion (i.e., data elements which did not either include or exclude
all patients homogeneously) to ensure that only criteria leading to a discrimination
within the preselected cohort were used. Finally, data elements with no available
data were downranked, as an implementation of the data availability categorization
proposed by Wang et al.[27 ] The cut-off was set at a score value of 2, as this set of criteria achieved a higher
specificity than a score of 3. Based on the paper chart review, the effect was even
higher. The cut-off was not set at score value 1, as this would have been almost identical
to using the full set of criteria (54 out of 70).
The resulting sensitivity and specificity show that the cohort derived from our prioritized
subset of eligibility criteria is larger than the set of actually recruited patients.
The set includes false-positive patients but no false negatives. While this ensures
that no potential candidate was excluded during the prescreening step, the false-positive
candidates require additional manual inspection. This matches the observation regarding
false positives as a result of limited data availability from Löbe et al.[26 ]
Comparison of the eligibility criteria of the KATHERINE study with the inventories
published by Doods et al[24 ]
[25 ] ([Fig. 3 ]) showed only partial coverage, due to the structure of the inventories. While some
attributes are listed with generic labels (e.g., “verbatim drug name” for medications),
the actual trial eligibility criteria referred to specific substances (e.g. “Doxorubicin”).
The inventory structure could be considered inconsistent with regard to the fact that
laboratory findings are not grouped but given individually (e.g., “total cholesterol
in serum”). Given diagnoses yielded a very low availability both in the data warehouse,
as well as chart review, in comparison to the inventory. The study protocol referred
to a set of specific diseases as exclusion criteria (which were rare in the cohort),
whereas in the inventory covered the presence of any diagnosis in the dataset. We
also noted that disease-specific criteria (e.g., number of chemotherapy cycles) are
underrepresented in the inventories ([Supplementary Figs. S1 ] and [S2 ]; available in the online version). As the inventories were generated by analyzing
the frequency of criteria across a large set of studies, it follows that criteria
relevant across several diseases shared higher frequencies whereas disease-specific
criteria would not reach the required threshold for inclusion into the inventories.
In our dataset, disease-specific criteria had a higher relevance toward the selection
of a prioritized subset of criteria. It should be considered to extend the inventories
with disease group–specific modules.
Löbe et al, Trinczek et al, and Zhang et al[26 ]
[32 ]
[36 ] noted that clinical trial candidate identification and screening for recruitment
currently are very time-consuming manual tasks. In our project, manual chart review
of the full set of eligibility criteria in a base population of 106 patients took
32.5 hours, whereas the time spent for constructing and executing the data warehouse
query was 15.5 hours. Since query implementation efforts relate only to the number
of criteria implemented, whereas manual chart review relates to the size of the cohort,
the potential gains of electronic execution should increase with the size of the base
population.
Averitt et al compared cohort compositions of four landmark randomized controlled
trials (RCTs) with cohorts derived from routine clinical datasets[37 ] and found that even though identical eligibility criteria were rigorously applied,
baseline summary statistics varied between published results and EHR-derived datasets,
suggesting heterogeneity of treatment effects (HTE) and putting replicability, as
well as medical applicability, of RCT results on real-world cohorts into doubt. Among
other measures, Averitt et al propose to implement a more structured, codified documentation
of eligibility criteria to enhance replicability. While electronic recruitment support
could help to standardize application of eligibility criteria, data availability,
and selection of implementable/prioritized criteria could introduce biases of their
own.
Limitations
The KATHERINE trial chosen for this project has very narrow eligibility criteria,
resulting in a very small percentage of actually recruitable patients within the available
base population, also contributing to the low–positive predictive value of 0.1%. Potential
patients which matched the eligibility criteria but may have declined to participate
in the study were not taken into account. Also, the project was performed only at
a single academic hospital with data availability from the routine care process limited
mostly to inpatient care. This could negatively impact the applicability of the results
on trials with broader criteria and/or other types of hospitals. The selection of
the trial to be used in the project was constrained by the scope and collaborating
partners in the relevant EHR4CR work package. The analysis of data availability was
performed not against the full patient population of the hospital, but against a preselected
cohort matching basic criteria (breast cancer and neoadjuvant treatment) determined
from a separate documentation platform not included in the data warehouse. A full
paper chart review would not have been feasible on the full patient population. This
mandatory preselection step is in fact an integral part of the proposed approach for
determining the prioritized subset of eligibility criteria to implement for patient
recruitment support. Beyond review of the cited literature, no specific training was
applied for the staff carrying out the classification and scoring of the eligibility
criteria. Subjectivity cannot be ruled out for some of the classification decisions
(e.g., comprehensibility). Whether the selected cut-off of score value 2 for inclusion
of criteria can be generalized to other trials needs to be confirmed. In this project,
only structured data elements in the clinical data warehouse were examined, and specifically
no natural language processing (NLP) approaches were performed to extract additional
data from narrative text (e.g., discharge letters). While NLP is increasingly being
applied on English-language datasets, it is not yet broadly implemented for German-language
EHRs.
Conclusion
Patient recruitment support for clinical trials based on electronic health records
is a topic of continuing interest. The prescreening step has been identified as the
focal point of establishing efficient recruitment support, yet no systematic process
for identifying a prioritized subset of eligibility criteria has yet been published.
Our proposed approach facilitates a data-driven selection of items based on their
relevance to the trial, the actual availability of data in the EHR and the resulting
discriminatory power of the chosen criteria. Apart from streamlining implementation
of electronic recruitment support, the approach could also be leveraged during the
protocol design, as well as site selection/feasibility phase. While the increasing
availability of structured EHR data provides an opportunity for secondary use in the
context of clinical trials, the quality of eligibility criteria in study protocols
with regard to their consistency and interpretability remains an important issue that
needs to be addressed. Annotation with standardized terminologies, inventories of
commonly used criteria (including disease-specific aspects) and possibly even reusable
databases of criteria[38 ] could be leveraged to simplify the implementation of electronic recruitment support.
With the current implementation of large-scale secondary use infrastructures, like
the German Medical Informatics Initiative (MII)[39 ] or the Swiss Personalized Health Network (SPHN),[40 ] harmonized platforms are currently becoming available that will also facilitate
patient recruitment support. Within the MII, the Medical Informatics in Research and
Care in University Medicine (MIRACUM) consortium pursues patient recruitment support
as a primary use case,[41 ] providing an infrastructure for a multicentric implementation and evaluation of
the approach presented in this paper.
Clinical Relevance Statement
Clinical Relevance Statement
Inclusion of patients in clinical trials is an integral part of, but not limited to,
academic medicine and can contribute to certification criteria (e.g., in comprehensive
cancer centers). Leveraging real-world data to support the recruitment process addresses
the need of hospitals to optimize the execution of clinical trials.
Multiple Choice Questions
Multiple Choice Questions
How should eligibility criteria for clinical trials be formulated?
Correct Answer : The correct answer is option b. Eligibility criteria for clinical trials should
be formulated to be selective (i.e., contain criteria that can be applied to derive
an eligible subset of probands) and atomic in the sense of limiting each criterion
to a single attribute. Complex criteria (containing several attributes) should be
decomposed into sets of atomic criteria. Redundancy should be avoided, including cases
in which a criterion appears both in the inclusion section (e.g., patients with M0
status), as well as in a negated form, in the exclusion section (e.g., patients with
M1 status). While it is desirable to have computable eligibility criteria (i.e., being
able to electronically derive from EHR data), in many cases criteria need physician
interpretation (e.g., whether a patient can be expected to comply to the study protocol).
Eligibility criteria should be formulated concisely, avoiding verbosity.
How can real-world data (RWD) support the execution of clinical trials?
RWD fully automates recruitment and execution of clinical trials
RWD obviates physician interpretation of eligibility criteria
RWD fully covers all attributes used for determining clinical trial eligibility
RWD can support adequate cohort size estimation and be used to select recruitment
candidates
Correct Answer : The correct answer is option d. RWD data typically covers only a subset of attributes
required for determining clinical trial eligibility, excluding, for example, data
not documented electronically, requiring physician interpretation or not documented
within a relevant timeframe. Applying a relevant subset of available, selective, and
prioritized data elements can be leveraged to achieve an adequate estimation of cohort
size and select candidates for prescreening. Final decisions toward inclusion or exclusion
into a trial need to be made by qualified personnel based not only on electronically
available data but also on data available from the paper chart or attributes acquired
during the screening process.
