Explanations of Statistical Concepts: Can they Penetrate the Haze of Bayes?

Summary

This experiment tested the ability of logical explanations to promote the use of statistical concepts. Sixty physicians in four experimental groups, matched for level of medical statistical training, were given a questionnaire that assessed their ability to rank the diagnosticity of various diagnostic test results. Each question was designed to test physician understanding of the impact of a particular statistical parameter, e.g., sensitivity and specificity, test-retest reliability. Logical explanations of the sources of test error increased the percentage of answers judged to be correct on the basis of published data. This was confirmed for a variety of questions. They tended to be less useful, however, when physicians expressed misinterpretations of statistical concepts or test patterns were not correctly perceived. The implications of these findings for medical education and for the design of computer-based explanation systems are clarified.

In diesem Experiment wurde die Fähigkeit logischer Erklärungen zur Förderung der Anwendung statistischer Begriffe untersucht. Sechzig Ärzten mit gleichem Niveau medizinisch-statistischer Ausbildung in vier Studiengruppen wurde ein Fragebogen gegeben, der ihre Fähigkeit zur Wertung der diagnostischen Macht verschiedener diagnostischer Testergebnisse einschätzen sollte. Jede Frage war so formuliert, daß sich aus der Antwort ergeben sollte, inwieweit der Arzt die Wirkung eines bestimmten statistischen Parameters (z.B. Sensitivität und Spezifizität, Test-Retest-Zuverlässigkeit) verstehen konnte. Logische Erklärungen der Quellen von Testfehlern erhöhten den Prozentsatz der aufgrund veröffentlichter Daten für richtig erachteten Antworten. Dies wurde für eine Vielfalt von Fragen bestätigt. Sie erwiesen sich jedoch als weniger nützlich, wenn Ärzte Fehlinterpretationen statistischer Begriffe zum Ausdruck brachten oder wenn Testmuster nicht richtig erkannt wurden. Die Folgerungen dieser Feststellungen für die ärztliche Ausbildung und für die Planung computerunterstützter Erklärungssysteme werden geklärt.

Key-Words:

Diagnostic Tests - Decision-making - Psychological Studies of Clinical Judgement - Computer-based Consultation Systems - Artificial Intelligence

Schlüssel-Wörter:

Diagnostische Tests - Entscheidungsfindung - psychologische Studien über klinische Beurteilung - computerunterstützte Konsultationssysteme - künstliche Intelligenz

^* Dr. Politser is recipient of Research Career Development Award LM 00080 from the National Library of Medicine. This research also was supported in part by Grants LM 04132, LM 04086 and LM 03366 from the National Library of Medicine and Grant HS 04726 from the National Center for Health Services Research.

• REFERENCES

• 1 Ajzen I. Intuitive Theories of Events and the Effects of Base-Rate Information on Prediction. J. Person. soc. Psych 35 1977; 303-314.
• 2 Bar-Hillel M. The Base Rate Fallacy in Probability Judgements. Acta Psychol 44 1980; 211-233.
• 3 Berwick D. M, Fineberg H. V, Weinstein M. D. When Doctors Meet Numbers. Presented at the Second Annual Meeting of the Society for Medical Decision Making Washington: D. C; 1980
• 4 Bleich H. L. Computer-based Consultation. Electrolyte and Acid-base Disorders. Amer. J. Med 53 1972; 285-291.
• 5 Bourne Jr. L. E, Guy D. E. Learning Conceptual Rules. II. The Role of Positive and Negative Instances. J. exp. Psych 77 1968; 488-194.
• 6 Casscells W, Schoenberger A, Gray-boys T. B. Interpretation by Physicians of Clinical Laboratory Results. New Engl. J. Med 299 1978; 999-1001.
• 7 Chapman L. J, Chapman J. P. Illusory Correlation as an Obstacle to the Use of Valid Psychodiagnostic Signs. J. abnorm. Psych 74 1969; 271-280.
• 8 Chase W. G. Simon, H. A.: Perception in Chess. Cogn. Psychol 4 1973; 55-61.
• 9 Christensen-Szalanski J. J, Bushyhead J. B. Physicians' Misunderstanding of Normal Findings. Med. Decis. Mak 3 1983; 169-175.
• 10 Detmer D. E, Fryback D. G, Gassner K. Heuristics and Biases in Medical Decision Making. J. med. Educ 53 1978; 682-683.
• 11 Eddy D. M. Screening for Cancer: Theory, Analysis and Design. Englewood Cliffs N. J; Prentice-Hall: 1980
• 12 Einhorn H. J, Kleinmuntz D. N, Klein-muntz B. Linear Regression and Process-tracing Models of Judgment. Psychol. Rev 86 1979; 465-485.
• 13 Erdman H. A Computer Consultation Program to Help Non-psychiatric Physicians Choose Treatments for Depression. Ph. D. Dissertation, Dept. of Industrial Engineering; University of Wisconsin.:
• 14 Feinstein A. R. The Haze of Bayes, the Aerial Palaces of Decision Analysis, and the Computerized Ouija Board. Clin. Biostat 21 1976; 482-496.
• 15 Golding S. L, Rorer L. G. Illusory Correlation and Subjective Judgement. J. abnorm. Psych 80 1972; 249-260.
• 16 Gorry G, Kassirer J, Essig A, Schwartz W. Decision Analysis as the Basis for Comper-aided Management of Acute Renal Failure. Amer. J. Med 55 1973; 473-484.
• 17 Greenland P, Muschlin A. I, Griner P. F. Discrepancies between Knowledge and Use of Diagnostic Studies in Asymptomatic Patients. J. med. Educ 54 1979; 863-869.
• 18 Griner P. F, Mayewski R. J, Mushline A. I, Greenland P. Selection and In-terpretation of Diagnostic Tests and Procedures. Ann. int. Med 94 1981; 553-600.
• 19 Hovland C. I, Weiss W. Transmission of Information Concerning Concepts through Positive and Negative Instances. J. exp. Psych 45 1953; 175-182.
• 20 Jacquez J. A, Norusis M. J. The Importance of Symptom Non-independence. In F. T. DeDombal and F. Gremy (Eds): Diagnosis, Decision Making and Medical Care. Amsterdam: North-Holland Publ. Co; 1976
• 21 Kahneman D. Tversky, A.: Judgement Under Uncertainty: Heuristics and Biases. Science 184 1974; 1124-1131.
• 22 Komaroff A. L. Algorithms and the "Art" of Medicine. Amer. J. Public Hlth 72 1982; 10-12.
• 23 Lopes L. Doing the Impossible: A Note on Induction and the Experiences of Ran-domness. Department of Psychology; University of Wisconsin: 1980
• 24 Ludwig D, Heilbronn D. The Design and Testing of a New Approach to Computer-aided Differential Diagnosis. Meth. Inform. Med 22 1983; 156-166.
• 25 Politser P. E. Decision Analysis and Clinical Judgement: A Re-evaluation. Med. Dec. Mak 1 1982; 368-389.
• 26 Politser P. E. The Evaluation of Repeated Medical Tests: Logical and Statistical Considerations. Ph.D. Dissertation; The University of Michigan: 1982
• 27 Politser P. E. Reliability, Decision Rules, and the Value of Repeated Test. Med. Dec. Mak 2 1982; 47-69.
• 28 Slovic P, Lichtenstein S. Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgement. Org. Behav. Hum. Perform 6 1971; 649-744.
• 29 Smedslund J. The Concept of Correlation. Scand. J. Psych 4 1963; 165-174.
• 30 Swensson R. G. A Two-stage Detection Model Applied to Skilled Visual Search by Radiologists. Percept. Psychophys 27 1080; 11-16.
• 31 Thorngate W. Efficient Decision Heuristics. Behav. Sei 25 1980; 219-225.
• 32 Wagenaar W. A. Appreciation of Conditional Probabilities in Binary Sequences. Acta Psychol 34 1970; 348-356.
• 33 Wallis J. W, Shortliffe E. H. Explanatory Power for Medical Expert Systems: Studies in the Representation of Causal Relationships for Clinical Consultations. Meth. Inform. Med 21 1982; 127-136.