Thromb Haemost 2022; 122(04): 570-577
DOI: 10.1055/a-1525-7220
Stroke, Systemic or Venous Thromboembolism

Machine Learning to Predict Outcomes in Patients with Acute Pulmonary Embolism Who Prematurely Discontinued Anticoagulant Therapy

Damián Mora
1   Department of Internal Medicine, Hospital Virgen de la Luz, Cuenca, Spain
José A. Nieto
1   Department of Internal Medicine, Hospital Virgen de la Luz, Cuenca, Spain
Jorge Mateo
2   Neurobiological Research Group, Institute of Technology, Universidad de Castilla-La Mancha, Cuenca, Spain
Behnood Bikdeli
3   Cardiovascular Medicine Division, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
4   Yale/YNHH Center for Outcomes Research and Evaluation, New Haven, Connecticut, United States
5   Cardiovascular Research Foundation (CRF), New York, New York, United States
6   Clinic of Angiology, University Hospital Zurich, Zurich, Switzerland
7   Center for Thrombosis and Hemostasis, University Hospital Mainz, Mainz, Germany
Javier Trujillo-Santos
8   Department of Internal Medicine, Hospital General Universitario Santa Lucía, Universidad Católica de Murcia, Murcia, Spain
Silvia Soler
9   Department of Internal Medicine, Hospital Olot i Comarcal de la Garrotxa, Gerona, Spain
Llorenç Font
10   Department of Haematology, Hospital de Tortosa Verge de la Cinta, Tarragona, Spain
11   Faculty of Medicine, University Cardiology Clinic, Skopje, Republic of Macedonia
Manuel Monreal
12   Department of Internal Medicine, Hospital Germans Trias i Pujol, Badalona, Barcelona, Spain
13   Department of Medicine, Universidad Católica de Murcia, Murcia, Spain
the RIETE Investigators › Author Affiliations
Funding The sponsors of RIETE had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication. B.B was supported by the National Heart, Lung, and Blood Institute, National Institutes of Health (NIH), through grant number T32 HL007854. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Dr. Bikdeli reports that he has been a consulting expert (on behalf of the plaintiff) for litigation related to a specific type of IVC filters. The current study is the idea of the investigators and has not been performed at the request of a third party.


Background Patients with pulmonary embolism (PE) who prematurely discontinue anticoagulant therapy (<90 days) are at an increased risk for death or recurrences.

Methods We used the data from the RIETE (Registro Informatizado de Pacientes con Enfermedad TromboEmbólica) registry to compare the prognostic ability of five machine-learning (ML) models and logistic regression to identify patients at increased risk for the composite of fatal PE or recurrent venous thromboembolism (VTE) 30 days after discontinuation. ML models included decision tree, k-nearest neighbors algorithm, support vector machine, Ensemble, and neural network [NN]. A “full” model with 70 variables and a “reduced” model with 23 were analyzed. Model performance was assessed by confusion matrix metrics on the testing data for each model and a calibration plot.

Results Among 34,447 patients with PE, 1,348 (3.9%) discontinued therapy prematurely. Fifty-one (3.8%) developed fatal PE or sudden death and 24 (1.8%) had nonfatal VTE recurrences within 30 days after discontinuation. ML-NN was the best method for identification of patients experiencing the composite endpoint, predicting the composite outcome with an area under receiver operating characteristic (ROC) curve of 0.96 (95% confidence interval [CI]: 0.95–0.98), using either 70 or 23 variables captured before discontinuation. Similar numbers were obtained for sensitivity, specificity, positive predictive value, negative predictive value, and accuracy. The discrimination of logistic regression was inferior (area under ROC curve, 0.76 [95% CI: 0.70–0.81]). Calibration plots showed similar deviations from the perfect line for ML-NN and logistic regression.

Conclusion The ML-NN method very well predicted the composite outcome after premature discontinuation of anticoagulation and outperformed traditional logistic regression.

* A full list of the RIETE investigators is given in [ Supplementary Appendix A ] .

Supplementary Material

Publication History

Received: 03 November 2020

Accepted: 08 June 2021

Accepted Manuscript online:
09 June 2021

Article published online:
13 July 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany