Keywords factor VIII - hemophilia A - machine learning - predictive modeling - turoctocog alfa
pegol
Introduction
Prophylaxis is considered the standard of care for the prevention and management of
bleeding in patients with hemophilia A.[1 ]
[2 ] Extended half-life (EHL) recombinant factor VIII (rFVIII) molecules were developed
to offer reduced dosing frequency and higher factor activity levels than standard
half-life FVIII molecules.[1 ] Switching patients to prophylaxis with an EHL rFVIII molecule can provide meaningful
improvements in health-related quality of life.[3 ] However, there is significant interpatient variability in response to rFVIII treatments,
which may relate to the patient's age, body mass, bleeding phenotype, genotypic variation,
ABO blood group, physical activity, and joint status, among other variables.[1 ]
[4 ]
[5 ]
[6 ]
Identifying patient characteristics that predict long-term outcomes could help to
inform clinical decisions for prophylaxis optimization. Annualized bleeding rate (ABR)
is an established outcome measure of prophylaxis efficacy as it correlates with long-term
joint destruction.[7 ]
[8 ] Previous attempts to predict and improve ABR have been focused on the use of FVIII
pharmacokinetic (PK) parameters. Collins et al established that time per week spent
at FVIII activity below 1 IU/dL was associated with increased total bleeds.[9 ] Valentino et al showed that peak FVIII levels, area under the curve, and time spent
per week with FVIII levels >20 IU/dL were linked to bleeding risk.[4 ] More recently, Tiede et al demonstrated that bleeding risk can change over time
and is influenced by factors independent to PK parameters, suggesting that other demographic
and clinical characteristics are required to predict long-term treatment response.[10 ] Notably, the aforementioned approaches to predict ABR relate to evidence using standard
half-life FVIII molecules for prophylaxis. Since then, guidelines have suggested to
target higher trough levels,[2 ] which is achievable with EHL rFVIII molecules. It is reasonable to reassess predictors
of long-term clinical response, as these may have changed with improvements in prophylaxis.
Predictive modeling involves the use of analytic techniques to predict clinical outcomes.
Machine learning is a powerful computational approach used to recognize patterns in
complex, multivariate datasets that include clinical variables and outcomes, enabling
the development of predictive modeling. A significant advantage of machine learning
techniques is their ability to handle highly variable datasets, collinearity, and
missing data. Importantly, they can be used to identify how much a variable contributes
to predicting a subsequent outcome. Rapid advances in machine learning techniques
have enabled the application of predictive modeling to data from randomized controlled
trials, showing significant promise in using patient and clinical characteristics
to identify variables that are strongly associated with a chosen treatment outcome
across a range of indications.[11 ]
[12 ]
[13 ]
N8-GP is an EHL human rFVIII product that has the potential to provide a simplified
prophylaxis treatment for patients with hemophilia A due to its fixed, body-weight-based
dosing regimen (50 IU/kg every 4 days [Q4D]).[14 ] The mean trough level observed during prophylaxis with N8-GP (turoctocog alfa pegol;
Esperoct, Novo Nordisk A/S, Bagsvaerd, Denmark) at 50 IU/kg every 4 days was 3 IU/dL.[15 ] The long-term safety and efficacy of N8-GP were demonstrated in the pivotal pathfinder
2 trial (NCT01480180), which is the only clinical trial of an EHL rFVIII molecule
where most patients received a fixed dosing regimen for prophylaxis to date. In most
other studies of EHL rFVIII molecules, patients received prophylaxis that was individualized
by investigators, or patients were stratified to receive different regimens based
on bleeding rate during a run-in phase.[16 ]
[17 ] For these studies, the application of machine learning techniques to identify predictive
patterns is difficult, because the intensity of therapy was, at least to some extent,
influenced by clinical outcomes. Due to its unique design, pathfinder 2 provides for
the first time an opportunity to identify predictive patterns in the setting of prophylaxis
with an EHL rFVIII molecule.
The purpose of this posthoc analysis was to develop a predictive model to identify
pre- and post-N8-GP prophylaxis variables that can act as predictors of clinical response
to fixed-dose prophylaxis by applying a machine learning framework to data from the
pathfinder clinical trial program.
Methods
Research Objectives
The objectives of this exercise included the following: (1) to identify pre-N8-GP
prophylaxis variables (including baseline characteristics) associated with an ABR
of ≤1 or >1 bleed/year at the end of the pathfinder 2 main phase; (2) to identify
post-N8-GP prophylaxis variables (including treatment-related variables, and patient
characteristics) associated with an ABR of ≤1 or >1 bleed/year at the end of the pathfinder
2 main phase. For the purposes of this analysis, the ABR threshold of 1 (rather than
“0”) was chosen as the outcome of interest, as the ability to identify and distinguish
early bleeds varies between patients, and it is not uncommon for some patients to
treat pain as an early bleed.
Data Source
Data from the pathfinder trials (pathfinder 1, NCT01205724; pathfinder 2, NCT01480180;
pathfinder 3, NCT01489111; pathfinder 5, NCT01731600; pathfinder 7, NCT02920398) were
investigated for the application of supervised machine learning for predictive modeling.
Of these, the pivotal pathfinder 2 trial[18 ]
[19 ]
[20 ] was selected for analysis due to the comprehensive array of variables from a sufficient
sample size of patients. Pathfinder 2 was a phase III, open-label trial investigating
long-term safety, PK, and efficacy of N8-GP used for prophylaxis (50 IU/kg Q4D) or
on-demand treatment in 186 previously treated patients (aged ≥12 years) with severe
hemophilia A. The full details of pathfinder 2 have been reported previously.[18 ]
[19 ]
[20 ]
Model Overview
Predictive modeling typically includes a predictive model learning phase, where data
are collected for input into the model, and a predictive model outcome period, where
the outcome of interest (ABR) is measured. The chosen clinical outcome of interest
for prediction was ABR of ≤1 versus >1 at the end of the outcome period. Multiple
predictive models using data from different time points in the study period were developed
to identify variables pre- and post-N8-GP prophylaxis that were associated with patients
achieving an ABR of ≤1 versus >1.
Each predictive model learning phase used data collected from one of five separate
time points in pathfinder 2. Pre-N8-GP prophylaxis models learned from data collected
at screening visit (“baseline model” ) or Visit 2 (prophylaxis initiation; “baseline and PK model” ). Post-N8-GP prophylaxis models learned from data collected between Visit 2 and 3
(i.e., up to 4 weeks postprophylaxis switch; “post-N8-GP 4-week prophylaxis model” ), Visit 4 (i.e., up to 8 weeks postprophylaxis switch; “post-N8-GP 8-week prophylaxis model” ), or Visit 5 (i.e., up to 12 weeks postprophylaxis switch; “post-N8-GP 12-week prophylaxis model” ), as well as the data collected at baseline ([Fig. 1 ]). The 12-week post-N8-GP prophylaxis time window was chosen as the maximum duration
of the predictive model learning phase as patients tend to be followed more closely
during this period by clinicians, where initial treatment response becomes apparent,
and this is often the earliest time that a change in treatment is considered.
Fig. 1 Predictive modeling methodology. a Multiple predictive models using data from different time points in the main phase
were developed to identify which variables were associated with patients achieving
an ABR of ≤1 versus >1 during the final stage of the outcome period. b ABR values were calculated for each model's outcome period, which varied between each
predictive model. ABR, annualized bleeding rate.
Within each model cohort, patients were classified by observed clinical response at
the end of the predictive model outcome period: ABR ≤1 versus >1. The predictive model
outcome period was the time between the end of the learning phase of each model's
time frame and the end of the pathfinder 2 main phase ([Fig. 1 ]). As individual ABRs were not reported at the end of the main phase, ABR values
were calculated based on the number of bleeds reported after each model's time point,
and during the final 365 days of the predictive model outcome period prior to the
last recorded site visit during the main phase for each patient. ABR values were calculated
for each patient for each model's outcome period, which varied between each predictive
model; patients received prophylaxis in the predictive model outcome period for approximately
15 months (median [min.–max. range]: 469 [107–747] days) between the baseline visit
and end of the main phase, and approximately 13 months (median [min.–max. range]:
385 [100–654] days) of prophylaxis between the Visit 5 time point and end of the main
phase. In patients where a complete 1-year outcome period was unavailable (n = 44/166 [baseline model]; n = 70/161 [post-N8-GP 12-week prophylaxis model]), ABR was calculated by scaling the
number of bleeds during the available predictive model outcome period. Patients on
Q4D prophylaxis who switched to twice-weekly during the trial due to insufficient
treatment response were included in the “ABR >1” group, irrespective of their ABR
outcome. For patients who did not complete the main phase, ABR was calculated at the
time of withdrawal.
Each predictive model included a cohort of patients from pathfinder 2 who received
consistent N8-GP prophylaxis (50 IU/kg Q4D). Data were excluded if the patient received
on-demand treatment or did not receive at least 90 days of prophylaxis exposure beyond
the model's learning time point. As such, the number of patients in each model varied
by time point, ranging from 166 patients in the pre-N8-GP prophylaxis models to 161
in the post-N8-GP 12-week prophylaxis model.
Variable Selection
Standardized predictor variables for predictive modeling were selected from the range
of patient and treatment characteristics collected at screening visit and after the
switch to N8-GP prophylaxis. Variables that were expected to have clinical relevance
were identified and reviewed by two clinicians before being selected for inclusion
in the predictive analyses ([Supplementary Table S1 ], available in the online version). Variable selection was based on clinical experience
and availability of patient data; no additional criteria for variable selection were
applied. Patient-reported outcomes such as EQ-5D were selected initially; however,
preliminary model analyses did not support their retention for predictive modeling
and hence these variables were excluded.
Predictive Modeling Methodology
Two machine learning techniques were used for predictive modeling: penalized logistic
regression and random forests (see the [Supplementary Material ], available in the online version). Additionally, interpretation methods for the
tree-based models were used to elucidate the underlying drivers of treatment response.
For explanation of predictions at the individual patient level, SHapley Additive exPlanations
(SHAP) was used to quantify the contribution that each variable brought to the prediction
made by the model. SHAP estimated how important each variable was by evaluating how
well the model performed with and without that variable.[21 ] In this analysis, SHAP values greater than zero implied an increased association
with an ABR ≤1; values less than zero implied an increased association with an ABR
>1. Global ranking of how each variable contributed to the predicted clinical outcome
at the group level was derived from mean absolute SHAP values.
Model Validation
To validate the predictive potential of this approach, a repeated nested cross-validation
strategy was implemented to alleviate “overfitting” and ensure the models could generalize
well to new data.[22 ] In this, each patient could contribute to the learning (training set) phase of model
development or act as a test patient (testing dataset or validation dataset), i.e.,
each patient was assigned a prediction from a corresponding predictive model. To facilitate
interpretation, the association between variables and subsequent clinical outcome
was described at the model level and individual patient level. A further description
of model validation is provided in the [Supplementary Material ] (available in the online version).
Performance Metrics
The performance metrics of each predictive model were assessed using the area under
the receiver operator curve (AUROC) to indicate how well variables collected at the
model's time point could be used to predict the patients' observed ABR outcome at
the end of the model outcome period. An AUROC of 1 indicated a perfect model and an
AUROC of 0.5 indicated a model that performs equivalently to random chance. The predictive
models were compared against a univariate benchmark to reference their performance.
The univariate benchmark was historical ABR for pre-N8-GP prophylaxis models and total
bleed count up to the model's time point for post-N8-GP prophylaxis models.
Results
Patient Characteristics
Of 175 patients initiated with N8-GP prophylaxis (50 IU/kg Q4D) in the pathfinder
2 main phase, a total of 166 received a consistent regimen and had sufficient follow-up
for inclusion in the baseline model. Patient demographics and baseline characteristics
for this cohort and the cohort of patients included in the post-N8-GP 12-week prophylaxis
model are reported in [Table 1 ].
Table 1
Patient demographics and characteristics for the pathfinder 2 main phase cohort at
screening visit (baseline) and Visit 5
Screening visit (baseline)
Visit 5
Total (N = 166)[a ]
ABR ≤1 (n = 87)
ABR > 1 (n = 79)
Total (N = 161)
ABR ≤1 (n = 85)
ABR > 1 (n = 76)
Age in years, mean (±SD)
30.5 (12.3)
31.3 (13.0)
29.7 (11.5)
30.5 (12.4)
31.0 (13.2)
29.8 (11.6)
Height in meters, mean (±SD)
1.76 (0.077)
1.77 (0.079)
1.74 (0.072)
1.76 (0.078)
1.77 (0.082)
1.74 (0.071)
BMI in kg/m2 , mean (±SD)
24.4 (3.9)
24.4 (3.8)
24.4 (4.0)
24.4 (3.8)
24.5 (3.8)
24.4 (3.9)
Historical ABR, mean (±SD)
12.9 (27.7)
7.3 (12.2)
19.0 (37.2)
13.1 (28.1)
8.8 (15.2)
17.9 (36.9)
von Willebrand factor in IU/mL, mean (±SD)
1.00 (0.40)
1.09 (0.42)
0.90 (0.35)
1.00 (0.41)
1.08 (0.43)
0.91 (0.36)
Previous prophylaxis regimen, n
141
78
63
136
76
60
Previous on-demand regimen, n
25
9
16
25
9
16
Treatment history (months on previous prophylaxis/on-demand regimen), mean (±SD)
117.8 (108.7)
113.7 (106.6)
122.4 (111.4)
119.5 (109.7)
121.4 (114.1)
117.4 (105.4)
Abbreviations: ABR, annualized bleeding rate; BMI, body mass index; SD, standard deviation;
vWF, von Willebrand factor.
a 14 of the 166 patients in this cohort switched to twice-weekly treatment and were
classified in the “ABR > 1” group.
ABR Outcome
Patients were classified by ABR at the end of the model outcome period (which ended
at approximately 15 months after the model's time point). In the baseline model's
cohort, 87 patients had an ABR ≤1, and 79 patients had an ABR >1. Details regarding
the duration of the predictive model learning phase and outcome period are reported
in [Table 2 ]. The majority (n = 122) of patients were exposed to prophylaxis for >1 year in the predictive model
outcome period.
Table 2
Duration and variables of the predictive model learning phase and predictive model
outcome period, and performance of the pre-N8-GP prophylaxis and post-N8-GP prophylaxis
predictive models
Predictive model
Pre-N8-GP prophylaxis models
Post-N8-GP prophylaxis models
Baseline model
Baseline and PK model
Post-N8-GP
4-week prophylaxis model
Post-N8-GP
8-week prophylaxis model
Post-N8-GP
12-week prophylaxis model
Patient count, n
166
166
165
162
161
Predictive model learning phase
Time points of the predictive model learning phase
Baseline
Baseline
Baseline and up to 4 weeks postswitch
Baseline and up to 8 weeks postswitch
Baseline and up to 12 weeks postswitch
Variables included for multivariable models
• 33 baseline variables
• 33 baseline variables
• 2 PK variables (trough, FVIII at 30 minutes)
• 33 baseline variables
• 2 PK variables: mean at Visit 3 (trough, FVIII at 30 minutes)
• Cumulative bleed count up to Visit 3 (up to 4 weeks postprophylaxis switch)
• 33 baseline variables
• 2 PK variables: mean at Visit 4 (trough, FVIII at 30 minutes)
• Cumulative bleed count up to Visit 4 (up to 8 weeks postprophylaxis switch)
• 33 baseline variables
• 2 PK variables: mean at Visit 5 (trough, FVIII at
30 minutes)
• Cumulative bleed count up to Visit 5 (up to 12 weeks postprophylaxis switch)
Predictive model outcome period
Duration of the predictive model outcome period, median (min.–max.) days
469 (107–747)
470 (107–747)
441 (94–714)
413 (115–686)
385 (100–654)
Patients with ABR ≤1 at the end of the outcome period, n
87
87
85
80
85
Patients with ABR >1 at the end of the outcome period, n
79
79
80
82
76
Model performance: single-variable benchmark
AUROC for historical ABR
0.5821
0.5821
–
–
–
AUROC for cumulative bleed count until visit
–
–
0.6588
0.7059
0.7480
Model performance: multivariable models
Penalized logistic regression, AUROC (95% CI)
0.636
(0.628–0.644)
0.672
(0.664–0.681)
0.689
(0.680–0.698)
0.707
(0.700–0.715)
0.724
(0.716–0.731)
Random forest, AUROC (95% CI)
0.608
(0.597–0.619)
0.622
(0.612–0.631)
0.690
(0.684–0.697)
0.758
(0.750–0.766)
0.785
(0.778–0.792)
Abbreviations: ABR, annualized bleeding rate; AUROC, area under the receiver operator
curve; CI, confidence interval; FVIII, factor VIII; PK, pharmacokinetic.
Note: Performance evaluation of the predictive models was assessed using AUROC. AUROC
values were rated as the following: AUROC of 0.5: model performs no better than random;
AUROC 0.5 to <0.6: poor performance; AUROC 0.6 to <0.7: moderate performance; AUROC
0.7 to <0.8: good performance; AUROC 0.8 to <0.9: excellent performance; AUROC 0.9
to <1.0: outstanding performance; AUROC of 1.0: perfect performance; interpretation
of AUROC values is subjective and influenced by clinical context. The single-variable
benchmark for screening and Visit 2 is based on thresholding historical ABR reported
at the screening visits. The benchmark for Visit 3 to Visit 5 is based on thresholding
the count of bleed events observed since the screening visits.
Performance of Predictive Models
The performance of the predictive models and single-variable benchmarks are reported
in [Table 2 ]. Univariate benchmark models were included to compare minimum performance against
performance of the pre- and post-N8-GP prophylaxis models developed by supervised
learning methods.
For the baseline model, an AUROC of 0.636 and 0.608 was reported for the penalized
logistic regression and random forests, respectively, indicating “moderate” performance
of this model for predicting the observed ABR outcome at the end of the outcome period.
Model performance was improved in the baseline and PK model with the addition of PK
measures, as indicated by higher AUROC values (0.672 and 0.622, respectively). Both
pre-N8-GP prophylaxis models demonstrated improved performance compared with the single-variable
benchmark (AUROC = 0.5821).
The post-N8-GP 12-week prophylaxis model was the most performant, with AUROC values
of 0.724 and 0.785 for the penalized logistic regression and random forests, respectively,
indicating “good” performance of this model for predicting the observed ABR outcomes.
This model demonstrated marginally better performance to the single-variable benchmark
of total bleed count until the model's time point (AUROC = 0.748). Model performance
increased incrementally over time from the post-N8-GP 4-week prophylaxis model to
the post-N8-GP 12-week prophylaxis model.
Model Interpretability and SHAP Values
SHAP values were used to quantify the contribution of each variable to the prediction
made by the model. [Fig. 2 ] depicts individual predictions from three patient cases in the baseline model. The
overall prediction for a patient (model output value) and the confidence in that prediction
are related to the sum of the underlying contributions made by each of their variables.[23 ]
Fig. 2 Individual predictions in the baseline model for (A ) a patient with a strong negative outcome prediction, (B ) a patient with an uncertain prediction, and (C ) a patient with a strong positive outcome prediction. This plot reports local interpretability
of individual patient predictions from three patient cases in the baseline model.
The model outputs a score between 0 and 1. The “base value” of 0.524 is the percentage
of patients with outcome ABR ≤1. The “model output value” for an individual patient
is the sum of the base value and the SHAP value contributions of all individual variables.
A model output value of less than the base value results in a prediction of ABR >1,
and greater than the base value results in a prediction of ABR ≤1. Variables that
increase the base value and predict ABR ≤1 are in pink , and visual size shows the magnitude of the effect. Variables that decrease the base
value and predict ABR >1 are in blue . ABR, annualized bleeding rate; SHAP, Shapley Additive Explanations.
Predictive Variables in the pre-N8-GP Prophylaxis Models
The local interpretability for each predictor was grouped to interpret the overall
effect of each variable on the model, which facilitates insight into associations,
but not causality. When ranked by importance, SHAP analysis for global interpretability
identified that the von Willebrand factor (vWF) level and historical ABR were the
most impactful clinically relevant baseline variables for predicting observed ABR
([Fig. 3A ]). Mean absolute SHAP values in [Fig. 4A ] illustrate the global importance of the variable and [Fig. 3A ] demonstrates the local explanation summary, demonstrating the direction of relationship
between the variable and outcome. Baseline vWF had an asymmetric distribution of SHAP
values, with extreme negative variable values influencing model predictions more than
extreme positive values.
Fig. 3 Global interpretability of (A ) the baseline model and (B ) the post-N8-GP 12-week prophylaxis model. Plot (A) reports variable importance and
variable effect. The three highest performing clinically relevant variables in the
baseline model are ranked according to mean absolute SHAP values, and shown as distributions
across individual patients. Each dot indicates a single patient's SHAP value; the
rank on the y -axis is determined by the mean absolute contribution of the variable to the model's
output; the position on the x -axis indicates the SHAP value; the coloring indicates the value of the variable (e.g.,
high vWF level in red , low vWF level in blue ). Mean absolute SHAP values for global interpretation of variable importance in the
baseline model are reported in [Fig. 4 ]. Plot (B) reports SHAP values for each patient extracted from the random forest
model using data collected at baseline/screening visit. Each dot indicates a single
patient's SHAP value; coloring indicates ABR outcome during the end of the outcome
period for each patient-level SHAP value. The red line indicates the variable threshold for positive and negative SHAP values. For both
plots, instances with SHAP values greater than zero correspond to patients with variable
values that influence the model toward predictions of ABR ≤1, whereas instances with
SHAP values less than zero correspond to patients with variable values that influence
the model toward predictions of ABR >1. ABR, annualized bleeding rate; SHAP, Shapley
Additive Explanations; vWF, von Willebrand factor.
Fig. 4 Mean absolute SHAP values for global interpretation of variable importance in the
(A ) baseline model and (B ) the post-N8-GP 12-week prophylaxis model. The figure depicts mean absolute SHAP
values for the 10 most informative variables in (A) the baseline model and (B) the
post-N8-GP 12-week prophylaxis model for predicting observed ABR outcomes during the
end of the outcome period. The SHAP value represents the amount a variable contributes
to the model output. Error bars report standard error of the mean. ABR, annualized
bleeding rate; SHAP, Shapley Additive Explanations.
SHAP analysis of variables at a patient level identified that patients with a threshold
vWF level <0.8 IU/mL or historical ABR >5 demonstrated negative SHAP values, indicating
an increased association with an ABR >1. Patients with a higher vWF level >0.8 IU/mL
or lower historical ABR <5 demonstrated positive SHAP values, indicating an increased
association with an ABR ≤1 ([Fig. 3B ]). An increase in the vWF level beyond 0.8 IU/mL was not associated with increasing
odds of the patient having an ABR ≤1, as indicated by the plateauing SHAP values;
however, decreasing levels of vWF below the 0.8 IU/mL threshold correlated with decreasing
SHAP values, indicating a stronger association with patients reporting an ABR >1.
The impact of height on predictions was most pronounced for the tallest patients (>1.8
m), who demonstrated positive SHAP values, indicating an increased association with
an ABR ≤1.
Predictive Variables in the Post-N8-GP Prophylaxis Models
Computation of SHAP values for global interpretability of the predictive model demonstrated
that cumulative count of treated bleeds (i.e., “total bleed count”) up to 12 weeks
post-prophylaxis switch, baseline vWF level, and mean FVIII at 30 minutes (Visit 5)
were the most important clinically relevant variables for predicting observed ABR
([Fig. 5A ]). Total bleed count up to 12 weeks postprophylaxis switch was the most informative
variable for predicting observed ABR, and markedly more informative than baseline
vWF level and mean FVIII at 30 minutes (Visit 5), as demonstrated by higher mean absolute
SHAP values (0.089, 0.033, and 0.030, respectively; [Fig. 4B ]). The performance of cumulative bleed count up to 12 weeks was “good” (AUROC = 0.748)
when assessed in isolation as a single-variable benchmark.
Fig. 5 (A ) Global interpretability of the post-N8-GP 12-week prophylaxis model and (B ) SHAP values for pharmacokinetic variables in the post-N8-GP 12-week prophylaxis
model. a Post-N8-GP prophylaxis variables. b von Willebrand factor level at baseline (pre-N8-GP prophylaxis). Plot (A) reports
variable importance and variable effect. The three highest performing clinically relevant
variables in the post-N8-GP 12-week prophylaxis model are ranked according to mean
absolute SHAP values, and shown as distributions across individual patients. Each
dot indicates a single patient's SHAP value; the rank on the y -axis is determined by the mean absolute contribution of the variable to the model's
output; the position on the x -axis indicates the SHAP value (positive values are predictive of lower ABR, and negative
values are predictive of ABR >1); the coloring indicates the value of the variable
(e.g., higher value in red , lower value in blue ). Mean absolute SHAP values for global interpretation of variable importance in the
post-N8-GP 12-week prophylaxis model are reported in [Fig. 4 ]. Plot (B) reports SHAP values for each patient extracted from the random forest
model using data collected at 12 weeks post-N8-GP prophylaxis switch (for the pharmacokinetic
variables). Each dot indicates a single patient's SHAP value; coloring indicates ABR
outcome during the end of the outcome period for each patient-level SHAP value. The
red line indicates the variable threshold for positive and negative SHAP values. For both
plots, instances with SHAP values greater than zero correspond to patients with variable
values that influence the model toward predictions of ABR ≤1, whereas instances with
SHAP values less than zero correspond to patients with variable values that influence
the model toward predictions of ABR >1. ABR, annualized bleeding rate; FVIII, factor
VIII; SHAP, Shapley Additive Explanations.
SHAP analysis for individual patient-level variable importance ranking identified
that a higher mean trough level (>3%) and a higher FVIII at 30 minutes postdose (>125%)
were associated with a patient achieving an ABR ≤1 ([Fig. 5B ]). FVIII at 30 minutes postdose demonstrated a wider distribution of SHAP values
compared with the trough level, indicating that FVIII at 30 minutes was more informative
for predicting ABR in the model. At the patient level, SHAP analysis demonstrated
that patients with a bleed count of zero universally had positive SHAP values (associated
with a patient achieving an ABR ≤1), whereas patients with a bleed count greater than
zero were universally shown to have negative SHAP values (i.e., encourage a prediction
of ABR >1).
Discussion
This posthoc analysis applied supervised machine learning techniques to data from
pathfinder 2 to develop predictive models to identify which pre- and post-N8-GP prophylaxis
variables act as predictors of clinical response to fixed-dose N8-GP prophylaxis.
The most performant predictive model was the post-N8-GP 12-week prophylaxis model,
which demonstrated that total bleed count during the initial 12-week period postswitch
to N8-GP prophylaxis was the most informative variable in the model for predicting
ABR at the end of the outcome period (which ended a median of approximately 13 months
after the model's time point, i.e., long-term response to N8-GP prophylaxis), followed
by baseline vWF level and mean FVIII at 30 minutes postdose. When assessed as a univariate
benchmark, total bleed count up to 12 weeks was as performant as the post-N8-GP 12-week
prophylaxis model, demonstrating the importance of clinical observation for predicting
long-term outcomes.
The performance of the baseline model was “moderate” (AUROC = 0.636), and only marginally
improved compared with the single-variable benchmark (historical ABR; AUROC = 0.5821).
The model indicated that the vWF level was the most impactful baseline variable for
predicting observed ABR. Patients who responded well to prophylaxis were those with
higher baseline vWF levels (>0.8 IU/mL), which is consistent with previous studies
demonstrating that higher vWF levels correlate with longer FVIII half-life and higher
FVIII activity between doses that may subsequently lead to a reduced ABR.[16 ]
[24 ] The model identified a linear relationship between low vWF and higher bleeds, which
plateaued beyond 0.8 IU/mL. The models identified that patients with greater height
(>1.8 m) reported positive SHAP values that were associated with an ABR ≤1. Although
taller patients may report lower body mass index (BMI) values, which is associated
with reduced clearance,[25 ] BMI and body weight were included as variables in the analysis and not found to
be informative predictors of ABR. Taller patients in this cohort may instead represent
a group of patients who were possibly more athletic, or had different levels of activity;
the association between a taller height and lower ABR is currently unclear. Overall,
the “moderate” performance (AUROC = 0.636) of the baseline model indicated that baseline
variables were limited in their performance to accurately predict ABR; as such, the
association between a taller height and lower ABR may be a spurious finding. The limited
performance of the baseline model confirms the dominant influence of the prophylaxis
regimen on ABR outcomes.
The post-N8-GP 12-week prophylaxis model was the most performant and assessed by AUROC
as “good” (AUROC = 0.785). The “good” (AUROC = 0.785) performance of this model indicates
that the 12-week observation window (learning phase) postswitch to N8-GP prophylaxis
is a sufficient period of time to monitor patients and determine whether a change
in treatment or regimen may be required, or investigations for other causative factors.
The post-N8-GP 12-week prophylaxis model included total bleed count; as a single-variable
benchmark, total bleed count up to 12 weeks was more performant than all variables
included in the baseline model (AUROC of 0.75 vs. 0.64, respectively). The post-N8-GP
12-week prophylaxis model and total bleed count univariate benchmark model demonstrate
the importance of bleeding phenotype-adjusted tailoring of prophylaxis over baseline
characteristics for predicting long-term ABR.
Previous studies investigating predictors of ABR identified that endogenous factor
levels, adherence, BMI, primary dosing regimen, inhibitor development, presence of
arthropathy, and intensity of physical activity as variables that can influence bleeding
rates.[1 ]
[4 ]
[26 ]
[27 ]
[28 ]
[29 ]
[30 ] In the present analysis, the post-N8-GP 12-week prophylaxis model demonstrated that
total bleed count up to 12 weeks postprophylaxis switch, baseline vWF level, and mean
FVIII at 30 minutes postdose were the most important variables for predicting observed
ABR with N8-GP prophylaxis. The mean absolute SHAP values demonstrated that the total
bleed count was substantially more informative than all other baseline characteristics
(vWF level) and PK measures (mean FVIII at 30 minutes and mean trough level), demonstrating
the importance of clinical observation for predicting long-term outcomes. The “good”
performance (AUROC = 0.748) of cumulative bleed count as a univariate benchmark model
is in line with clinical observations, whereby patients who bleed less in the initial
prophylaxis period of a trial are more likely to have fewer bleeds later in the trial.
These results suggest that bleeding events following prophylaxis initiation are indicative
of long-term outcomes and should prompt a review of the prophylactic regimen or investigation
for local pathology that may be contributing to an excess of bleeds. In routine clinical
practice, while prophylaxis tends to be adjusted based on treatment response, there
is no agreed period for review or an agreed cut-off for number of bleeds before deciding
on a treatment change. For the first time, this posthoc analysis confirms that a 12-week
treatment period is adequate to initiate treatment review in the event of bleeds,
for intensification of prophylaxis or investigation into other contributing factors.
The post-N8-GP 12-week prophylaxis model recognized that a higher trough level (>3%)
and FVIII at 30 minutes (representing peak FVIII activity of N8-GP[15 ]; >125%) were both associated with an ABR ≤1. Similarly, the literature reports that
maintaining a higher FVIII trough level and achieving a higher FVIII peak provides
increased protection from joint and nonjoint bleeding, and subsequent improvements
in ABR.[4 ]
[10 ]
[30 ] FVIII at 30 minutes and trough level may act as surrogate markers for N8-GP PK area
under the curve and decay of the curve. SHAP analysis indicated that the FVIII level
at 30 minutes postdose was more informative than the trough level for predicting observed
ABR. This result suggests that achieving the observed peak threshold of >125% is associated
more strongly with a higher PK area under the curve and an ABR ≤1 than achieving a
trough level >3%. However, this result may also reflect the quality of data collected,
whereby values for FVIII level at 30 minutes are less anomalous than the trough level,
and may therefore act as a more informative measure.
Previous investigations into predicting ABR outcomes using standard half-life FVIII
focused on the association of FVIII PK measurements with ABR.[4 ]
[9 ] In this analysis with an EHL FVIII molecule, the inclusion of PK measurements improved
model performance, as demonstrated by higher AUROC values for the baseline and PK
model. However, the contribution of mean FVIII at 30 minutes and mean trough level
to the predictive power of the post-N8-GP 12-week prophylaxis model was substantially
lower than the total bleed count, as demonstrated by lower mean SHAP values (0.030
and 0.016 vs. 0.089, respectively). Although PK data may be used to inform decisions
and individualize treatment in clinical practice (e.g., dose adjustments),[31 ] the results reported in the present analysis suggest that using clinical observations
of total bleed count alone during the initial 12-week prophylaxis period (univariate
assessment) in the context of fixed-dose prophylaxis with a high trough level is as
good and potentially more informative to predict long-term prophylaxis response. The
results confirm the value of a simplified, fixed-dose prophylaxis regimen for patients,
with possible adaptation of treatment according to bleed count in the initial weeks
postinitiation, rather than based on the residual clotting factor activity levels.[32 ]
Limitations of this analysis included the modest sample size of patients (n = 166), which was subject to selection bias, whereby patients included were those
who remained in the study for longer and could have been more likely to be compliant
or be motivated to remain in the study due to observed improvement. The analysis was
based on main phase data from pathfinder 2 only; the performance of the predictive
models may benefit from a longer observation time with refined outcomes of interest.
Data were not available for joint status, which is considered a clinically relevant
variable for phenotypic assessment prior to treatment initiation in hemophilia A patients.[33 ] Data included in the analyses were for patients receiving N8-GP prophylaxis Q4D,
and hence there was limited scope to compare findings against other regimens, doses,
or treatments. Due to the modest sample size of patients, validation was performed
using nested cross-validation (i.e., “internal validation”). Although nested cross-validation
is an acceptable method for validating a predictive model, the use of a separate dataset
for external validation is more robust for confirming model performance. The dataset
included anomalies in trough level data, and a proportion of trough level values was
below the detection limit threshold (0.045 IU/mL). FVIII at 30 minutes and FVIII trough
level were the only PK parameters from pathfinder 2 for which there were sufficient
data for inclusion in the predictive models. Machine learning for predictive modeling
was applied to an EHL PEGylated FVIII molecule; although the association between clinically
relevant variables and predicted ABR outcomes is unlikely to be specific to N8-GP,
results cannot be readily extrapolated to standard half-life FVIII molecules, or those
with different half-life extension technology.
To refine the performance and value of the predictive models in future analyses, results
should be replicated in data from other hemophilia clinical trials, as well as the
real-world setting where patient behavior may be more variable, and where a broader
set of variables may be available. As the performance of each predictive model improved
with the inclusion of more variables, the availability of additional clinically relevant
variables such as joint status, physical activity, and intensity of previous prophylaxis
regimen may help to further increase model performance. Additionally, the availability
and inclusion of other PK parameters, such as area under the curve, should be applied
to future machine learning analyses to further investigate the role of PK. The ability
to predict ABR as a continuous value or range of values (rather than as a binary outcome
of ABR ≤1 vs. ABR >1) may affect the model, as the occurrence of a single bleed event
would have a large impact on the defined outcome.
Conclusion
Machine learning for predictive modeling is a novel approach to analyzing data from
hemophilia clinical trials. In this posthoc analysis, applying supervised machine
learning techniques for predictive modeling to data from the pathfinder 2 trial demonstrated
that cumulative bleed count in the 12 weeks postprophylaxis switch was the most informative
variable for predicting observed ABR at the end of the outcome period (approximately
13 months later), and probably more informative than PK assessments and baseline characteristics.
This outcome confirms observations in clinical practice in which patients who bleed
less in the initial period posttreatment switch are more likely to have fewer long-term
bleeding events. Additionally, it places greater emphasis on phenotype-adjusted tailoring
of prophylaxis according to bleed response, as well as investigations for other mechanisms
of bleeding. The results confirm the value of simplified fixed-dose prophylaxis with
adaptations based on both clinical outcomes and PK parameters. Furthermore, this analysis
reports the first use of machine learning techniques applied to data from a hemophilia
clinical trial, with the aim to identify results that can be applied in the clinic.
The analysis should be highlighted as an example for the use of artificial intelligence
techniques applied to data from future clinical trials in hemophilia for any therapeutic
modality.
What is known about this topic?
The most recent iteration of the World Federation of Hemophilia guidelines[2 ] recommends a higher trough level for prophylaxis, which can be achieved using extended
half-life (EHL) factor VIII (FVIII) molecules, and the use of personalized prophylaxis
plans.
Previous studies have attempted to predict bleeding risk and annualized bleeding rate
(ABR) in people with hemophilia A utilizing FVIII pharmacokinetic (PK) parameters,
and reported an association between bleeding risk and peak FVIII level, area under
the curve, and time per week with FVIII activity below 1 IU/dL.
However, bleeding risk can change over time and may be influenced by factors independent
to PK parameters, suggesting that other demographic and clinical characteristics are
required to predict long-term treatment response to prophylaxis.
What does this paper add?
To our knowledge, this is the first analysis reporting the application of supervised
machine learning techniques for predictive modeling to data from a hemophilia clinical
trial (namely the pathfinder 2 trial in the setting of prophylaxis with the EHL recombinant
FVIII molecule N8-GP).
Predictive modeling in the present analysis identified that cumulative bleed count
up to 12 weeks postswitch to N8-GP prophylaxis, baseline von Willebrand factor, and
mean FVIII at 30 minutes postdose were the most informative variables for predicting
ABR outcomes during the final year of the main phase (median of 385 days after the
model's time point) in patients who received fixed-dose prophylaxis with N8-GP.
Bleed count at 12 weeks post-N8-GP prophylaxis switch was more informative than PK
measures (trough level, mean FVIII at 30 minutes postdose) and baseline characteristics
for predicting long-term ABR, emphasizing the importance of bleeding phenotype-adjusted
tailoring of prophylaxis.
