Download PDF
Methods Inf Med 1981; 20(02): 80-96
DOI: 10.1055/s-0038-1635297
Original Article
Schattauer GmbH

The Measurement of Performance in Probabilistic Diagnosis IV. Utility Considerations in Therapeutics and Prognostics

DIE LEISTUNGSMESSUNG BEI DER WAHRSCHEINLICHKEITSDIAGNOSE — IV. NÜTZLICHKEITSERWÄGUNGEN IN THERAPEUTIK UND PROGNOSTIK
J. D. F. Habbema
1   From the Department of Public Health and Social Medicine, Erasmus University, Rotterdam, The Netherlands, and the Institute of Human Genetics, University of Copenhagen, Denmark
,
J. Hilden
1   From the Department of Public Health and Social Medicine, Erasmus University, Rotterdam, The Netherlands, and the Institute of Human Genetics, University of Copenhagen, Denmark
› Author Affiliations
Further Information

Publication History

Publication Date:
15 February 2018 (online)

    Permissions and Reprints

It is argued that it is preferable to evaluate probabilistic diagnosis systems in terms of utility (patient benefit) or loss (negative benefit). We have adopted the provisional strategy of scoring performance as if the system were the actual decision-maker (not just an aid to him) and argue that a rational figure of merit is given by the average loss which patients would incur by having the system decide on treatment, the treatment being selected according to the minimum expected loss principle of decision theory.

A similar approach is taken to the problem of evaluating probabilistic prognoses, but the fundamental differences between treatment selection skill and prognostic skill and their implications for the assessment of such skills are stressed. The necessary elements of decision theory are explained by means of simple examples mainly taken from the acute abdomen, and the proposed evaluation tools are applied to Acute Abdominal Pain data analysed in our previous papers by other (not decision-theoretic) means. The main difficulty of the decision theory approach, viz. that of obtaining good medical utility values upon which the analysis can be based, receives due attention, and the evaluation approach is extended to cover more realistic situations in which utility or loss values vary from patient to patient.

Es wird argumentiert, daß wahrscheinlichkeitstheoretische Diagnosesysteme vorzugsweise nach Nützlichkeit (Vorteil für den Patienten) oder Schaden (Nachteil für den Patienten) ausgewertet werden sollten. Die Autoren benutzen die provisorische Strategie der Leistungsbewertung so, als ob das System der eigentliche Entscheidungsfinder sei (und nicht nur eine Hilfe für diesen), und argumentieren, daß eine vernünftige Maßzahl für den Nutzen durch den durchschnittlichen Schaden gegeben wird, den Patienten erleiden würden, wenn das System über ihre Behandlung entscheiden würde, wobei die Behandlung so ausgewählt wird, daß der Schaden im Sinne der Entscheidungstheorie minimiert wird.

Ähnlich wird das Problem der Auswertung wahrscheinlichkeitstheoretischer Prognosen angegangen; aber die grundlegenden Unterschiede zwischen der Geschicklichkeit in der Auswahl der Behandlung und der Prognosestellung und ihre Folgen für die Bewertung solcher Fähigkeiten werden hervorgehoben. Die erforderlichen Elemente der Entscheidungstheorie werden an einfachen Beispielen, vorwiegend aus dem Gebiet des akuten Abdomens, erläutert, und die vorgeschlagenen Auswertungsinstrumente werden auf bereits in früheren Arbeiten mit anderen Methoden nicht entscheidungstheoretischer Natur analysierte Daten über akute Bauchschmerzen angewandt. Dem Hauptproblem bei Anwendung des entscheidungstheoretischen Ansatzes, nämlich dem, gute medizinische Nützlichkeitswerte für die Analyse zu erhalten, wird gebührende Aufmerksamkeit gezollt, und der Auswertungsansatz wird erweitert, um realistischere Situationen zu erfassen, bei denen die Werte für Nutzen und Schaden von Patient zu Patient variieren.

Key-Words:

Computer-aided Diagnosis - Probabilistic Diagnosis - Probabilistic Prognostication - Treatment Selection - Utility Assessment - Decision Theory - Prognosis - Scoring Rules - Evaluation of Clinical Skills

Schlüssel-Wörter:

Computerunterstützte Diagnose - wahrscheinlichkeitstheoretische Diagnose - wahrscheinlichkeitstheoretische Prognosestellung - Wahl der Behandlung - Nützlichkeitsbewertung - Entscheidungstheorie - Prognose - Bewertungsregeln - Bewertung klinischer Fähigkeiten
 

  • References

  • 1 Arrchison J. Statistical Problems of Treatment Allocation. J. roy. statist. Soc A 1970; 133: 206-239.
    CrossrefPubMedGoogle Scholar
  • 2 Alpérovitch A, Knill-Jones R. P, Le Minor M, Lacaine F, Poynard T. Assessment of the Costs of Diagnostic Errors: Methodology and Preliminary Results. In Alpérovitch A, de Dombal F. T, Grémy F. (Eds): Evaluation of Efficacy of Medical Action Amsterdam: North-Holland Publ. Co; 1979: 167-177.
    Google Scholar
  • 3 Asselain B, Derouesné C, Salamon R, Bernadet M, Grémy F. The Concept of Utility in Medical Decision Aid: Example of an Application. In Shires D. B, Wolf H. (Eds): MEDINFO 77 Amsterdam: North-Holland Publ. Co; 1977: 123-125.
    Google Scholar
  • 4 Bjerregaard B, Brynitz S, Holst-Christensen J, Kalaja E, Lund-Kristensen J, Hilden J, de Dombal F. T, Horrocks J. C. Computer-aided Diagnosis of the Acute Abdomen: A System from Leeds Used on Copenhagen Patients. In de Dombal F. T, Grémy F. (Eds): Decision Making and Medical Care: Can Information Science Help? Amsterdam: North-Holland Publ. Co; 1976: 165-174.
    Google Scholar
  • 5 Bjerregaard B, Brynitz S, Holst-Christensen J, Jess P, Kalaja E, Knutssøn B, Lund-Kristensen J. Differentialdiagnostik hos patienter med akutte mavesmerter. Ugeskr. Læg 1978; 140: 830-834.
    PubMedGoogle Scholar
  • 6 Bunker J. P, Barnes B. A, Mosteller F. (Eds) Costs, Risks, and Benefits of Surgery. New York: Oxford Univ. Press; 1977
    Google Scholar
  • 7 Card W. I. Computer-Assisted Diagnosis and Pattern Recognition. Proc. roy. Soc. (Lond.) B 1973; 184: 421-432.
    CrossrefPubMedGoogle Scholar
  • 8 Card W. Computer-Aided Diagnosis — Theory and Practice (Chairman’s Introduction). In Anderson J, For-sythe J. M. (Eds): MEDINFO 74, 2 Amsterdam: North-Holland Publ. Co; 1974: 1067-1068.
    Google Scholar
  • 9 Card W. I, Good I. J. The Estimation of the Implicit Utilities of Medical Consultants. Math. Biosci 1970; 6: 45-54.
    CrossrefPubMedGoogle Scholar
  • 10 Card W. I, Rusinkiewicz M, Phillips C. I. Utility Estimation of a Set of States of Health. Meth. Inform. Med 1977; 16: 168-175.
    PubMedGoogle Scholar
  • 11 Fishburn P. C. Utility Theory. Manag. Sci 1968; 14: 335-378.
    CrossrefPubMedGoogle Scholar
  • 12 Habbema J. D. F, Hilden J, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis — I. The Problem, Descriptive Tools, and Measures Based on Classification Matrices. Meth. Inform. Med 1978; 17: 217-226.
    PubMedGoogle Scholar
  • 13 Habbema J. D. F, Hilden J, Bjerregaard B. The Measurement of Performance in Probalilistic Diagnosis — V. General Recommendations. Meth.Inform.Med 1981; 20: 97-100.
    PubMedGoogle Scholar
  • 14 Habbema J. D. F, van der Maas P. J. Decision Theory in Clinical Medicine: An Appraisal of its Uses, its Assumptions, and its Limits. In Girelli-Bruni E. (Edit.): Teoría delle decisioni in medicina: Aspetti sociali, ideologici e metodologici della statistica in medicina Verona: Bertani Editore; 1980
    Google Scholar
  • 15 Hershey J. C. Consequence Evaluation in Decision Analytic Models of Medical Screening, Diagnosis, and Treatment. Meth. Inform. Med 1974; 13: 197-203.
    PubMedGoogle Scholar
  • 16 Hilden J. The Concept of Medical Usefulness in Clinical Chemistry and the Difficulty of Applying it to Questions of Research Policy. Scand. J. clin. Lab. Invest. 40 Suppl 1980; 155: 31-46.
    PubMedGoogle Scholar
  • 17 Hilden J, Habbema J. D. F, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis. II. Trustworthiness of the Exact Values of the Diagnostic Probabilities. Meth. Inform. Med 1978; 17: 227-237.
    PubMedGoogle Scholar
  • 18 Hilden J, Habbema J. D. F, Bjerregaard B. The Measurement of Performance in Probabilistic Diagnosis. III. Methods Based on Continuous Functions of the Diagnostic Probabilities. Meth. Inform. Med 1978; 17: 238-246.
    PubMedGoogle Scholar
  • 19 IEEE Trans. Systems, Man, and Cybernetics: Special Issue on Behavioral Decision-Making SMC— 1977; 7: 309-405.
    PubMed
  • 20 Johnson N. L, Kotz S. Distributions in Statistics, Continuous Univariate Distributions — 2. . Boston: Houghton Mifflin; 1970. now New York: Wiley.;
    Google Scholar
  • 21 J roy. Coll. Phycns London Special Issue on Statistical Methods for Clinical Decision Making. 1975; 9: 191-269.
    PubMedGoogle Scholar
  • 22 Keeney R. L, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: Wiley; 1976
    Google Scholar
  • 23 Komaroff A. L. The Variability and Inaccuracy of Medical Data. Proc. IEEE 1979; 67: 1196-1207.
    CrossrefPubMedGoogle Scholar
  • 24 Krischer J. P. An Annotated Bibliography of Decision Analytic Applications to Health Care. Operations Res 1980; 28: 97-113.
    CrossrefPubMedGoogle Scholar
  • 25 Lancet: Editorial. 1979; I: 809-810.
    PubMed
  • 26 Lindley D. V. Costs and Utilities. In de Dombal F. T, Grémy F. (Eds): Decision Making and Medical Care: Can Information Science Help? Amsterdam: North-Holland Publ. Co; 1976: 101-112.
    Google Scholar
  • 27 Matheson J. E, Winkler R. L. Scoring Rules for Continuous Probability Distributions. Managmt Sci 1976; 22: 1087-1096.
    PubMedGoogle Scholar
  • 28 Mcneil B. J, Pauker S. G. The Patient’s Role in Assessing the Value of Diagnostic Tests. Radiology 1979; 132: 605-610.
    CrossrefPubMedGoogle Scholar
  • 29 Mcneil B. J, Weichselbaum R, Pauker S. G. Fallacy of the Five-Year Survival in Lung Cancer. New Engl. J. Med 1978; 299: 1397-1401.
    CrossrefPubMedGoogle Scholar
  • 30 Murphy A. H. A Note on the Utility of Probabilistic Predictions and the Probability Score in the Cost-Loss Ratio Decision Situation. J. appl. Meteorol 1966; 5: 534-537.
    CrossrefPubMedGoogle Scholar
  • 31 Murphy A. H. The Value of Weather Forecasts: Some Theoretical Results. In Murphy A. H, Williamson D. L. (Eds): Weather Forecasting and Weather Forecasts: Models, Systems, and Users Boulder Colorado: National Center for Atmospheric Research; 1977: 579-598.
    Google Scholar
  • 32 Murphy A. H, Winkler R. L. Credible Interval Temperature Forecasting: Some Experimental Results. Mon. Weath. Rev. (Amer. Meteor. Soc.) 1974; 102: 784-794.
    PubMedGoogle Scholar
  • 33 New Engl J. Med. Issue on Decision Making in Health Care. 1975; 293: 211-244.
    PubMedGoogle Scholar
  • 34 Patrick E. A, Stelmack F. P, Shen L.Y.L. Review of Pattern Recognition in Medical Diagnosis and Consulting Relative to a New System Model. IEEE Trans. Syst. Man Cybern SMC— 1974; 4: 1-16.
    PubMedGoogle Scholar
  • 35 Pearl J. An Economic Basis for Certain Methods of Evaluating Probabilistic Forecasts. Int. J. Man-Machine Studies 1978; 10: 175-183.
    PubMedGoogle Scholar
  • 36 Pliskin J. S, Beck C. H. Decision Analysis in Individual Clinical Decision Making: A Real-World Application in Treatment of Renal Disease. Meth. Inform. Med 1976; 15: 43-46.
    PubMedGoogle Scholar
  • 37 Ransohoff D. F, Feinstein A. R. Is Decision Analysis Useful in Clinical Medicine? Yale J. Biol. Med 1976; 49: 165-168.
    PubMedGoogle Scholar
  • 38 Savage L. J. The Foundation of Statistics. New York: Dover; 1972. Originally, New York: Wiley; 1954
    Google Scholar
  • 39 Smets P, Willems J, Talmon J, Demaertelaer V, Kornreich F. Methodology for the Comparison of Various Diagnostic Procedures. Biometrie-Praximetrie 1975; 15: 89-122.
    PubMedGoogle Scholar
  • 40 Staël von Holstein C.A.S, Murphy A. H. The Family of Quadratic Scoring Rules. Mon. Weath. Rev. (Amer. Meteor. Soc.) 1978; 106: 917-924.
    PubMedGoogle Scholar
  • 41 Szolovitz P, Pauker S.G. Computers and Clinical Decision Making: Whether, How, and For Whom? Proc. IEEE 1979; 67: 1224-1226.
    CrossrefPubMedGoogle Scholar
  • 42 Taylor T. R, Aitchison J, Parker L. S, Moore M. F. Individual Differences in Selecting Patients for Regular Hæmodialysis. Brit. med. J 1975; II: 380-381.
    PubMedGoogle Scholar
  • 43 Wulff H. R. Probabilities and Value Judgements in Clinical Decision-Making. In Alpérovitch A, de Dombal F. T, Grémy F. (Eds): Evaluation of Efficacy of Medical Action Amsterdam: North-Holland Publ. Co; 1979: 199-207.
    Google Scholar