The inter-rater reliability of the Pulmonary Embolism Severity Index

Montserrat Fraga; Patrick Taffé; Marie Méan; Olivier Hugli; Stéphanie Witzig; Gérard Waeber; Drahomir Aujesky

doi:10.1160/TH10-07-0426

Thrombosis and Haemostasis, Table of Contents

Thromb Haemost 2010; 104(06): 1258-1262
DOI: 10.1160/TH10-07-0426

New Technologies, Diagnostic Tools and Drugs

Schattauer GmbH

The inter-rater reliability of the Pulmonary Embolism Severity Index

Authors

Montserrat Fraga

¹Division of General Internal Medicine University of Lausanne, Lausanne, Switzerland
Patrick Taffé

²Institute of Social and Preventive Medicine, University of Lausanne, Lausanne, Switzerland
Marie Méan

¹Division of General Internal Medicine University of Lausanne, Lausanne, Switzerland
Olivier Hugli

³Emergency Department, University of Lausanne, Lausanne, Switzerland
Stéphanie Witzig

¹Division of General Internal Medicine University of Lausanne, Lausanne, Switzerland
Gérard Waeber

¹Division of General Internal Medicine University of Lausanne, Lausanne, Switzerland
Drahomir Aujesky

⁴Division of General Internal Medicine, Bern University Hospital, Bern, Switzerland

Abstract

Summary

The Pulmonary Embolism Severity Index (PESI) is a validated clinical prognostic model for patients with acute pulmonary embolism (PE). Our goal was to assess the PESI’s inter-rater reliability in patients diagnosed with PE. We prospectively identified consecutive patients diagnosed with PE in the emergency department of a Swiss teaching hospital. For all patients, resident and attending physician raters independently collected the 11 PESI variables. The raters then calculated the PESI total point score and classified patients into one of five PESI risk classes (I-V) and as low (risk classes I/II) versus higher-risk (risk classes III-V). We examined the inter-rater reliability for each of the 11 PESI variables, the PESI total point score, assignment to each of the five PESI risk classes, and classification of patients as low versus higher-risk using kappa (κ) and intra-class correlation coefficients (ICC). Among 48 consecutive patients with an objective diagnosis of PE, reliability coefficients between resident and attending physician raters were > 0.60 for 10 of the 11 variables comprising the PESI. The inter-rater reliability for the PESI total point score (ICC: 0.89, 95% CI: 0.81–0.94), PESI risk class assignment (κ: 0.81, 95% CI: 0.66–0.94), and the classification of patients as low versus higher-risk (κ: 0.92, 95% CI: 0.72–0.98) was near perfect. Our results demonstrate the high reproducibility of the PESI, supporting the use of the PESI for risk stratification of patients with PE.

Keywords

Inter-rater reliability - Pulmonary Embolism Severity Index

Full Text

References

References
1 Aujesky D, Obrosky DS, Stone RA. et al. Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med 2005; 172: 1041-1046.
2 Aujesky D, Roy PM, Le Manach CP. et al. Validation of a model to predict adverse outcomes in patients with pulmonary embolism. Eur Heart J 2006; 27: 476-481.
3 Jimenez D, Yusen RD, Otero R. et al. Prognostic models for selecting patients with acute pulmonary embolism for initial outpatient therapy. Chest 2007; 132: 24-30.
4 Donze J, Le Gal G, Fine MJ. et al. Prospective validation of the Pulmonary Embolism Severity Index. A clinical prognostic model for pulmonary embolism. Thromb Haemost 2008; 100: 943-948.
5 Wicki J, Perrier A, Perneger TV, Bounameaux H, Junod AF. Predicting adverse outcome in patients with acute pulmonary embolism: a risk score. Thromb Haemost 2000; 84: 548-552.
6 Sanchez O, Trinquart L, Caille V. et al. Prognostic Factors for Pulmonary Embolism: The PREP Study, A Prospective Multicenter Cohort Study. Am J Respir Crit Care Med 2009; 181: 168-173.
7 Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. J Am Med Assoc 1997; 277: 488-494.
8 Cohen JA. Coefficient of agreement for nominal scales. Ed Psychol Meas 1960; 20: 37-46.
9 Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968; 70: 213-220.
10 Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 420-428.
11 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174.
12 Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85: 257-268.
13 Chan CM, Woods C, Shorr AF. The validation and reproducibility of the Pulmonary Embolism Severity Index. J Thromb Haemost 2010; 8: 1509-1514.
14 Nordenholz KE, Naviaux NW, Stegelmeier K. et al. Pulmonary embolism risk assessment screening tools: the interrater reliability of their criteria. Am J Emerg Med 2007; 25: 285-290.
15 Wolf SJ, McCubbin TR, Feldhaus KM. et al. Prospective validation of Wells Criteria in the evaluation of patients with suspected pulmonary embolism. Ann Emerg Med 2004; 44: 503-510.
16 Runyon MS, Webb WB, Jones AE. et al. Comparison of the unstructured clinician estimate of pretest probability for pulmonary embolism to the Canadian score and the Charlotte rule: a prospective observational study. Acad Emerg Med 2005; 12: 587-593.
17 Jiménez D, Aujesky D, Moores L. et al. Simplification of the Pulmonary Embolism Severity Index for prognostication in patients with acute symptomatic pulmonary embolism. Arch Intern Med 2010; 170: 1383-1389.