Download PDF
Methods Inf Med 1978; 17(04): 217-226
DOI: 10.1055/s-0038-1636685
Original Article
Schattauer GmbH

The Measurement of Performance in Probabilistic Diagnosis

I. The Problem, Descriptive Tools, and Measures Based on Classification MatricesDIE LEISTUNGSMESSUNG BEI DER WAHRSCHEINLICHKEITSDIAGNOSE.I. DAS PROBLEM, DESKRIPTIVE VERFAHREN UND AUF KLASSIFIKATIONSMATRIZEN GEGRÜNDETE MESSGRÖSSEN
J. D. F. Habbema
1   (From the Department of Public Health and Social Medicine, Erasmus University, Rotterdam, The Netherlands, and the Institute of Human Genetics, University of Copenhagen, Denmark)
,
J. Hilden
1   (From the Department of Public Health and Social Medicine, Erasmus University, Rotterdam, The Netherlands, and the Institute of Human Genetics, University of Copenhagen, Denmark)
,
B. Bjerregaard
1   (From the Department of Public Health and Social Medicine, Erasmus University, Rotterdam, The Netherlands, and the Institute of Human Genetics, University of Copenhagen, Denmark)
› Author Affiliations
Further Information

Publication History

Publication Date:
19 February 2018 (online)

    Permissions and Reprints

Owing to the inherent uncertainty of diagnostic tasks, diagnostic advice should be offered in a probabilistic, rather than deterministic form. Since the late fifties a lot of effort has been invested in constructing probabilistic diagnosis rules. Much less has been done to devise rational tools for evaluating them ; conventional error rates reflect but one aspect of performance in a rather crude way. The aim of this paper and its successors is to offer a body of evaluation tools. After defining a general framework and stating its limitations we apply some graphical techniques to the acute abdominal pain data that will serve as illustration in the next papers as well: dot diagrams, the triangular diagram for the three-disease case, and three tabular representations based on categorization of the probabilities, viz. the usual (forced) classification matrix, from which various classification rates are read off; the classification matrix with doubt, in which uncertain and confident diagnoses are distinguished; and the exclusion matrix, which spots diseases that are judged improbable. Together these matrices give a good first impression of the behaviour of a probabilistic diagnosis system. The outlined techniques of later papers are needed for a more complete analysis.

Aufgrund der inhärenten Unsicherheiten diagnostischer Aufgaben sollte diagnostische Beratung besser in probabilistischer als in deterministischer Form geliefert werden. Seit den späten fünfziger Jahren hat man sich sehr um die Erstellung probabilistischer Diagnostikregeln bemüht. Viel weniger ist getan worden, um rationelle Hilfsmittel zu ihrer Bewertung ausfindig zu machen; die üblichen Fehlerquoten reflektieren nur einen Aspekt der Leistung, zudem in recht grober Weise. Das Ziel dieser Artikelreihe ist es, einen Satz von Bewertungsmethoden anzubieten. Nach Festlegung eines Begriffsrahmens und kurzer Diskussion seiner Grenzen wenden wir einige graphische und tabellarische Techniken auf Daten an, die aus der Kopenhagener Studie über akute Bauchschmerzen stammen und der Artikelreihe als durchgehendes Beispiel dienen : Punktdiagramme, das Dreieck-Diagramm für den Fall von drei Krankheiten und drei tabellarische Darstellungen, die auf Kategorisierung der Wahrscheinlichkeiten gegründet sind : die übliche (erzwungene) Klassifikationsmatrix, die Klassifikationsmatrix mit Verzichtsmöglichkeit, in der unsichere und sichere Diagnosen getrennt gezählt werden, und die Ausschlußmatrix, in der für unwahrscheinlich gehaltene Krankheiten aufgezählt werden. Zusammen geben diese Matrizen einen guten ersten Eindruck vom Verhalten eines probabilistischen Diagnosesystems. Für eine vollständigere Analyse werden allerdings die Techniken der beiden folgenden Aufsätze benötigt.

Key-Words:

Computer-aided Diagnosis - Uncertainty in Diagnosis - Probabilistic Diagnosis - Forced Classification Matrix - Classification Matrix with Doubt - Exclusion Matrix - Triangular Representation

Schlüssel-Wörter:

Computerunterstützte Diagnose - Unsicherheit bei der Diagnose - probabilistische Diagnose - erzwungene Klassifikationsmatrix - Klassifikationsmatrix mit Verzichtsmöglichkeit - Ausschlußmatrix - Dreiecksdarstellung
 

  • References

  • 1 Anderson J. A, Boyle J. A. Computer Diagnosis: Statistical Aspects. Brit. med. Bull 24 1968; 230-235.
    CrossrefPubMedGoogle Scholar
  • 2 Bjerregaard B, Brynitz S, Holst-Christensen J, Ka-laja E, Lund-Kristensen J, Hilden J, de Dombal F. T, Horrocks J. C. Computer-aided Diagnosis of the Acute Abdomen: a System from Leeds Used on Copenhagen Patients. In [6] 165-171.
    Google Scholar
  • 3 Bjerregaard B, Brynitz S, Holst-Christensen J, Jess P, Kalaja E, knutssøN B, Lund-Kristensen J. Diffe-rentialdiagnostik hos Patienter med Akutte Mavesmerter. Ugeskr. Laeg 140 1978; 830-834.
    PubMedGoogle Scholar
  • 4 Chow C. K. On Optimum Recognition Error and Reject Tradeoff. IEEE Trans. Inform. Theory IT 16 1970; 41-46.
    PubMedGoogle Scholar
  • 5 Documenta Geigy: Wissenschaftliche Tabellen Basle: J. R. Geigy, A. G.; 1960
  • 6 Dombal F. T, de Grémy F. (Eds) : Decision Making and Medical Care. Can Information Science Help ?. Amsterdam: North Holland Publ. Co; 1976
    Google Scholar
  • 7 Fragu P, Patois E, Huber C, Lellouch J. Original Approach of the Hyperthyroidism Diagnosis with Computer. In Anderson J, Forsythe J. M. (Eds) : MEDINFO 74, Vol. 2 Amsterdam: North-Holland Publ. Co; 1974: 559-564. 1068-1069.
    Google Scholar
  • 8 Habbema J. D. F. A Discriminant Analysis Approach to the Identification of Human Chromosomes. Biometrics 32 1976; 919-928.
    CrossrefPubMedGoogle Scholar
  • 9 Habbema J. D. F. Models for Diagnosis and Detection of Combinations of Diseases. In [6] 399-410.
    Google Scholar
  • 10 Habbema J. D. F, Hermans J, van der Bürgt A. T. Cases of Doubt in Allocation Problems. Biometrika 61 1974; 313-324.
    CrossrefPubMedGoogle Scholar
  • 11 Habbema J. D. F, Hermans J, Remme J. Data Analytical Methods in Discriminant Analysis : the Analysis of Posterior Probabilities. In Data Analysis and Informatics. Rocquencourt: IRLA; 1977: 211-221.
    Google Scholar
  • 12 Habbema J. D. F, Hilden J, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis. IV. Measures Based on Utility Considerations and General Recommendations. Meth. Inform. Med. (Under preparation.)
    Google Scholar
  • 13 Hermans J, Eggermont J. J, Hagedooren J, Odenthal D. W. Probabilistic Differential Diagnosis of Auditory Dysfunction on the Basis of Electrocochleography. Meth. Inform Med 14 1978; 87-95.
    PubMedGoogle Scholar
  • 14 Hilden J, Bjerregaard B. Computer-aided Diagnosis and the Atypical Case. In [6] 365-374.
    Google Scholar
  • 15 Hilden J, Habbema J. D. F, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis. II. Trustworthiness of the Exact Values of the Diagnostic Probabilities. Meth. Inform. Med 17 1978; 227-237.
    PubMedGoogle Scholar
  • 16 Hilden J, Habbema J. D. F, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis. III. Methods Based on Continuous Functions of the Diagnostic Probabilities. Meth. Inform. Med 17 1978; 238-246.
    PubMedGoogle Scholar
  • 17 Jennett B, Teasdale G, Braakman R, Minderhoud J, Knill-Jones R. Predicting Outcome in Individual Patients after Severe Head Injury. Lancet 1976; I: 1031-1034.
    PubMedGoogle Scholar
  • 18 J. ROY. Coll. Physicns London Special Issue on Statistical Methods for Clinical Decision Making. 9 1975; 191-269.
    PubMedGoogle Scholar
  • 19 Lachenbruch P. A, Mickey M. R. Estimation of Error Rates in Discriminant Analysis. Technometrics 10 1968; 1-11.
    CrossrefPubMedGoogle Scholar
  • 20 New Engl. J. Med Issue on Decision Making in Health Care. 293 1975; 211-244.
    PubMedGoogle Scholar
  • 21 Toronto A. F, Veasy G, Warner H. R. Evaluation of a Computer Program for Diagnosis of Congenital Heart Disease. Progr. Cardiov. Dis 5 1963; 362-377.
    CrossrefPubMedGoogle Scholar
  • 22 Vecchio T. J. Predictive Value of a Single Diagnostic Test in Unselected Populations. New Engl. J. Med 274 1966; 1171-1173.
    CrossrefPubMedGoogle Scholar
  • 23 Wagner G, Tautu P, Woiber U. Problems of Medical Diagnosis — A Bibliography. Meth. Inform. Med 17 1978; 55-74.
    PubMedGoogle Scholar