The Relative Accuracy of a Variety of Medical Diagnostic Programs

B. S. Todd; R. Stamper

doi:10.1055/s-0038-1635048

Methods of Information in Medicine, Table of Contents

Methods Inf Med 1994; 33(04): 402-416
DOI: 10.1055/s-0038-1635048

Original Article

Schattauer GmbH

The Relative Accuracy of a Variety of Medical Diagnostic Programs

B. S. Todd

¹The Programming Research Group, Oxford University Computing Laboratory, UK

,

R. Stamper

¹The Programming Research Group, Oxford University Computing Laboratory, UK

› Author Affiliations

Abstract

Abstract:

Acute abdominal pain is one of the most widely studied applications of computer-aided diagnosis. The usual approach is to apply Bayes’ theorem with the assumption of conditional independence (“independence Bayes”). We compared various approaches to designing diagnostic programs for abdominal pain of suspected gynaecological origin. The methods range from statistical to knowledge-based. All programs were evaluated using a database of 1,270 cases collected retrospectively. Our results suggest that in this application no significant improvement in accuracy can be made by taking interactions into account, either by statistical or by knowledge-based means; independence Bayes is near-optimal. As far as accuracy is concerned, there appears to be little point in pursuing knowledge-based approaches. However, the “nearest neighbours” method using a new metric appears to be at least as accurate as independence Bayes. We argue that the nearest neighbours method is more suitable than independence Bayes for clinical use because of greater accountability.

Keywords

Computer-aided Diagnosis - Nearest Neighbours - Abdominal Pain

Full Text

References

REFERENCES
1 De Dombal FT. The diagnosis of acute abdominal pain with computer assistance: worldwide perspective. Ann Chir 1991; 45: 273-7.
2 Adams ID, Chan M, Clifford PC. et al. Computer-aided diagnosis of acute abdominal pain: a multicentre study. BMJ 1986; 293: 800-4.
3 De Dombal FT, Leaper DJ, Horrocks JC, Staniland JR, McCann AP. Computer-aided diagnosis of acute abdominal pain. BMJ 1972; ii: 9-13.
4 Wellwood JM, Spiegelhalter DJ. Computers and the diagnosis of acute abdominal pain. Br J Hosp Med 1989; 41: 564-7.
5 Gunn AA. The acute abdomen: the role of computer-assisted diagnosis. Baillieres Clin Gastroenterol 1991; 05: 639-65.
6 Wellwood JM, Johannessen S, Spiegelhalter DJ. How does computer-aided diagnosis improve the management of acute abdominal pain?. Ann R Coll Surg Engl 1992; 74: 40-6.
7 Paterson-Brown S. Strategies for reducing inappropriate laparotomy rate in the acute abdomen. BMJ 1991; 303: 1115-8.
8 Paterson-Brown S, Vipond MN, Simms K, Gatzen C, Thompson JN, Dudley HAF. Clinical decision making and laparoscopy versus computer prediction in the management of the acute abdomen. Br J Surg 1989; 76: 1011-3.
9 Stonebridge PA, Freeland P, Rainey JB, Macleod DAD. Audit of computer-aided diagnosis of abdominal pain in accident and emergency departments. Arch Emerg Med 1992; 09: 271-3.
10 Sutton GC. Computer-aided diagnosis: a review. Br J Surg 1989; 76: 82-5.
11 Dixon JM, Elton RA, Rainey JB, Macleod DAD. Rectal examination in patients with pain in the right lower quadrant of the abdomen. BMJ 1991; 302: 386-8.
12 Norusis MJ, Jacquez JA. Diagnosis. II. Diagnostic models based on attribute clusters: a proposal and comparisons. Comput Biomed Res 1975; 08: 173-88.
13 Fryback DG. Bayes’s theorem and conditional nonindependence of data in medical diagnosis. Comput Biomed Res 1978; 11: 423-34.
14 Todd BS, Stamper R. Limits to diagnostic accuracy. Med Inf (Lond) 1993; 18: 255-70.
15 Hilden J. Statistical diagnosis based on conditional independence does not require it. Comput Biol Med 1984; 14: 429-35.
16 Russek E, Kronmal RA, Fisher LD. The effect of assuming independence in applying Bayes’ theorem to risk estimation and classification in diagnosis. Comput Biomed Res 1983; 16: 537-52.
17 Guppy KH, Detrano R, Abbassi N, Janosi A, Sandhu S, Froelicher V. The reliability of probability analysis in the prediction of coronary artery disease in two hospitals. Med Decis Making 1989; 09: 181-9.
18 Norusis MJ, Jacquez JA. Diagnosis. I. Symptom nonindependence in mathematical models for diagnosis. Comput Biomed Res 1975; 08: 156-72.
19 Nordyke R, Kulikowski CA, Kulikowski CW. A comparison of methods for the automated diagnosis of thyroid dysfunction. Comput Biomed Res 1971; 04: 374-89.
20 Croft DJ, Machol RE. Mathematical methods in medical diagnosis. Ann Biomed Eng 1987; 02: 69-89.
21 Titterington DM, Murray GD, Murray LS. et al. Comparison of discrimination techniques applied to a complex data set of head-injured patients. J R Statist Soc A 1981; 144: 145-75.
22 Orient JM. Evaluation of abdominal pain: clinicians’ performance compared with three protocols. South Med J 1986; 79: 793-9.
23 Gammerman A, Thatcher AR. Bayesian diagnostic probabilities without assuming independence of symptoms. Meth Inform Med 1991; 30: 15-22.
24 Fox J, Barber D, Bardhan KD. A quantitative comparison with rule-based diagnostic inference. Meth Inform Med 1980; 19: 210-5.
25 Heckerman DE, Horvitz EJ, Nathwani BN. Toward normative expert systems: Part I. The Pathfinder project. Meth Inform Med 1992; 31: 90-105.
26 De Dombal FT. Computer-aided decision support in clinical medicine. Int J Biomed Comput 1989; 24: 9-16.
27 Heckerman DE, Nathwani BN. An evaluation of the diagnostic accuracy of Pathfinder. Comput Biomed Res 1992; 25: 56-74.
28 Edwards FH, Davies RS. Use of a Bayesian algorithm in the computer-assisted diagnosis of appendicitis. Surg Gynecol Obstet 1984; 158: 219-22.
29 Cooper GF. A diagnostic method that uses causal knowledge and linear programming in the application of Bayes’ formula. Comput Meth Progr Biomed 1986; 22: 223-37.
30 Harris N, Spiegelhalter DJ, Bull K, Franklin RCG. Criticizing conditional probabilities in belief networks. In: Proceedings of the Fourteenth Annual Symposium on Computer Applications in Medical Care. Los Alamitos, CA: IEEE Computer Society Press; 1990: 805-9.
31 Seroussi B. The ARC & AURC Cooperative Group. Computer-aided diagnosis of acute abdominal pain when taking into account interactions. Meth Inform Med 1986; 25: 194-8.
32 Ebehart RC, Dobbins RW, Hutton LV. Neural network paradigm comparisons for appendicitis diagnoses. In: Proceedings of Fourth Annual IEEE Symposium on Com-puter-Based Medical Systems. Los Alamitos, CA: IEEE Computer Society Press; 1991: 298-304.
33 Emparanza JI, Aldamiz-Echevarria L, Perez-Yarza EG, Aldazabal P, Tovar J. Statistical approach to the acute abdomen diagnosis in children. In: Expert Systems and Decision Support in Medicine. Thirty-third Annual Meeting of the GMDS EFMI Special Topic Meeting. Peter L. Reichertz Memorial Conference. Berlin: Springer-Verlag; 1988: 131-5.
34 Todd BS, Stamper R. The formal design and evaluation of a variety of medical diagnostic programs. Technical Monograph PRG-109. Oxford University Computing Laboratory; 1993
35 Dickson JAS, Jones A, Telfer S, de Dombal FT. Acute abdominal pain in children. Scand J Gastroenterol Suppl 1988; 144: 43-6.
36 Telfer S, Fenyo G, Holt PR, de Dombal FT. Acute abdominal pain in patients over 50 years of age. Scand J Gastroenterol Suppl 1988; 144: 47-50.
37 Neapolitan RE. Probabilistic Reasoning in Expert Systems. New York: John Wiley; 1989
38 Lauritzen SL, Spiegelhalter DJ. Local computation with probabilities on graphical structures and their application to expert systems. J R Statist Soc B 1988; 50: 157-224.
39 Pauerstein CJ. Gynecological Disorders. Differential Diagnosis and Therapy. New York: Grune and Stratton; 1982
40 Chamberlain G. Contemporary Gynaecology. London: Butterworths; 1984
41 Whitfield CR. Dewhurst’s Textbook of Obstetrics and Gynaecology for Postgraduates, 4th ed. Oxford: Blackwell Scientific Publications; 1986
42 Cestnik B. Estimating probabilities: a crucial task in machine learning. In: Proceedings of the Ninth European Conference on Artificial Intelligence. (ECAI90). London: Pitman; 1990: 147-9.
43 Shortliffe EH. Computer-based Medical Consultations: MYCIN. New York: American Elsevier; 1976
44 Duda R, Gaschnig J, Hart P. Model design in the PROSPECTOR consultant system for mineral exploration. In: Michie D. ed. Expert Systems in the Micro-Electronic Age. Edinburgh: Edinburgh University Press; 1979: 153-67.
45 Felix WSWong, Leung KS, So YT. The recent development and evaluation of a medical expert system (ABVAB). Int J Biomed Comput 1990; 25: 223-9.
46 Spiegelhalter DJ, Knill-Jones RP. Statistical and knowledge-based approaches to clinical decision-support systems, with an application in gastroenterology. J R Statist Soc A 1984; 147: 35-77.
47 Heckerman DE, Horvitz EJ. The myth of modularity in rule-based systems. In: Lem-mer JF, Kanal LN. eds. Uncertainty in Artificial Intelligence 2. Amsterdam: North-Holland Publ Comp; 1988: 23-34.
48 Dan Q, Dudeck J. Certainty factor theory: its probabilistic interpretations and problems. Artif Intell Med 1992; 04: 21-34.
49 Todd BS, Stamper R, Macpherson PM. A probabilistic rule-based expert system. Int J Biomed Comput 1993; 33: 129-48.
50 Anderson JA. Logistic discrimination. In: Krishnaiah R, Kanal LN. eds. Handbook of Statistics 2. Amsterdam: North-Holland Publ Comp; 1982: 169-91.
51 Todd BS, Stamper R, Macpherson PM. The design and construction of a medical simulation model. Comput Meth Progr Biomed 1994; 42: 77-91.
52 Shortliffe EH, Buchanan B, Feigenbaum EA. Knowledge engineering for medical decision making: a review of computer-based clinical decision aids. Proc IEEE 1979; 67: 1207-24.
53 Macartney FJ. Diagnostic logic. BMJ 1987; 295: 1325-31.
54 Franklin RCG, Spiegelhalter DJ, Macartney FJ, Bull K. Evaluation of a diagnostic algorithm for heart disease in neonates. BMJ 1991; 302: 935-9.
55 Stamper R, Todd BS, Macpherson PM. A software engineering approach to the design of a medical expert system. In: Proceedings of the Fourth International Conference on Software Engineering and Knowledge Engineering (SEKE 92). Los Alamitos, CA: IEEE Computer Society Press; 1992: 341-8.
56 Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. Monterey: Wadsworth and Brooks/Cole; 1984
57 Cover TM, Hart PE. Nearest neighbour pattern classification. IEEE Trans Inf Theory 1967; 13: 21-7.
58 Todeschini T. k-nearest neighbour method: the influence of data transformations and metrics. Chemometrics Intell Lab Syst 1989; 06: 213-20.
59 Salzberg S. Distance metrics for instance-based learning. Lecture Notes Artif Intell 1991; 542: 399-408.
60 Bounds DG, Lloyd PJ, Mathew BG. A comparison of neural network and other pattern recognition approaches to the diagnosis of low back pain. Neural Networks 1990; 03: 583-91.
61 Baxt WG. Use of an artificial neural network for the diagnosis of myocardial infarction. Ann Intern Med 1991; 115: 843-8.
62 Maclin PS, Dempsey J, Brooks J, Rand J. Using neural networks to diagnose cancer. J Med Syst 1991; 15: 11-21.
63 Akay M. Noninvasive diagnosis of coronary artery disease using a neural network algorithm. Biol Cybern 1992; 67: 361-7.
64 Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature 1986; 323: 533-6.
65 Crichton NJ, Fryer JC, Spicer CC. Some points on the use of “independent Bayes” to diagnose acute abdominal pain. Stat Med 1987; 06: 945-59.
66 Crichton NJ, Hinde JP. Correspondence analysis as a screening method for indicants for clinical diagnosis. Stat Med 1989; 08: 1351-62.
67 Ohmann C, Kunneke M, Zaczyk R, Thon K, Lorenz W. Selection of variables using “independence Bayes” in computer-aided diagnosis of upper gastrointestinal bleeding. Stat Med 1986; 05: 503-15.
68 Chard T, Rubenstein EM. A model-based system to determine the relative value of different variables in a diagnostic system using Bayes theorem. Int J Biomed Comput 1989; 24: 133-42.
69 Mosteller F. Some statistical problems in measuring the subjective response to drugs. Biometrics 1952; 08: 220-6.
70 Hart A, Wyatt J. Connectionist models in medicine: an investigation of their potential. In: Proceedings of the Second European Conference on Artificial Intelligence in Medicine (AIME 89). Berlin: Springer-Verlag; 1989: 115-24.
71 Phillips S, Wiles J, Schwartz S. A comparison of three classification algorithms on the diagnosis of abdominal pains. In: Proceedings of the Second Australian Conference on Neural Networks (ACNN 91). Sydney University Press; 1991: 283-7.
72 Clark LA, Pregibon D. Tree-based models. In: Chambers JM, Hastie TJ. eds. Statistical models in S. Pacific Grove: Wadsworth and Brooks/Cole; 1992
73 Teach R, Shortliffe E. An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput Biomed Res 1981; 14: 542-58.
74 Young D, Chapman T, Poile C. Physician, reveal thyself. Br J Healthcare Comput 1990; 07: 16-21.
75 Stamper T, Todd BS, Macpherson PM. Case-based explanation for medical diagnostic programs, with an example in gynaecology. Meth Inform Med 1994; 33: 205-13.
76 Seroussi B. The ARC & AURC Cooperative Group. Comparison of several discrimination methods. Application to the acute abdomen. Lecture Notes Med Inf 1985; 28: 12-S.
77 Reggia JA, Perricone BT. Answer justification in medical decision support systems based on Bayesian classification. Comput Biol Med 1985; 15: 161-7.
78 De Dombal FT, Barnes S, Dallos V. et al. How should computer-aided decision-support systems present their predictions to the practising surgeon?. Theor Surg 1992; 07: 111-6.