Validation of ICTERUS, a Knowledge-based Expert System for Jaundice Diagnosis

Abstract:

The study aimed to describe an example of the assessment and validation of knowledge-based clinical expert systems. The paper focuses on ICTERUS, an expert system for jaundice diagnosis. It describes system design, the methodology applied for upgrading and validating the program, and the most important outcomes of the validation procedure. The clinical validation of the system on a very large European database (Euricterus Project) shows that diagnostic conclusions are reliable in about 70% of eligible cases. This figure appears acceptable for a system which provides decision support only on the basis of clinical data, assuming that the final decision is achieved under user responsibility. Expected biases, limitations and inconsistencies in the practical application of the system are discussed.

Keywords

Jaundice - Liver Diseases - Knowledge-based Expert System - System Validation - Diagnostic Accuracy

• REFERENCES

• 1 Chandrasekeran B. On evaluating AI systems for medical diagnosis. AI Magazine 1983; 4: 34-7.
• 2 Foster D, Buchanan B. Broadening our approach to evaluating medical information systems. Proc 15th SCAMC, Washington DC: 1991: 8-12.
• 3 Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Method Inform Med 1993; 32: 1-8.
• 4 Hilden J, Habbema DF. Evaluation of clinical decision-aids: more to think about. Med Inform 1990; 15: 275-84.
• 5 Miller PL, Sitting DF. The evaluation of clinical decision support systems: what is necessary versus what is interesting. Med Inform 1990; 15: 185-90.
• 6 Nguyen T, Perkins W, Laffey T. et al. Knowledge base verification. AI Magazine Summer 1987; 8: 69-75.
• 7 O’Neil M, Glowinski A. Evaluating and validating very large knowledge-based systems. Med Inform 1990; 15: 237-52.
• 8 Perrault L, Wiederhold G. System design and evaluation. In: Shortliffe et al (Eds). Med Inform. 1990: 151-77.
• 9 Swets JA. Measuring the accuracy of diagnostic systems. Science 1988; 240: 1285-93.
• 10 Friedman C, Wyatt J. Evaluation methods in medical informatics. New York: Springer Verlag; 1997
• 11 Van der Loo RP. Overview of published assessment and evaluation studies. In: Assessment and Evaluation of Information Technologies in Medicine. van Gennip EMSJ, and Talmon JL. Eds Amsterdam: IOS Press; 1995: 261-81.
• 12 Talmon JL, and Van der Loo RP. Literature on assessment of information technology and medical KBS evaluation: studies and methodologies. In: Assessment and Evaluation of Information Technologies in Medicine. van Gennip EMSJ, and Talmon JL. Eds. Amsterdam: IOS Press; 1995: 283-327.
• 13 Molino G, Molino F. Assessment and evaluation of knowledge-based expert systems for medical diagnosis. In: Assessment and Evaluation of Information. van Gennip EMSJ, and Talmon JL. Eds. Amsterdam: IOS Press; 1994: 151-66.
• 14 Lindberg G. Studies on diagnostic decision making in jaundice. Stockholm: Karolinska Institutes (Thesis); 1982: 1-60.
• 15 Matzen P, Malchow-Møller A, Hilden J. et al. Differential diagnosis of jaundice: a pocket diagnostic chart. Liver 1984; 4: 360-71.
• 16 Boom R, Gonzalez C, Fridman L. et al. Looking for “indicants” in the differential diagnosis of jaundice. Med Decis Making 1986; 6: 36-41.
• 17 Malchow-Møller A, Thomsen C, Matzen P. et al. Computer diagnosis in jaundice. Bayes’ rules founded on 1002 consecutive cases. J Hepatol 1986; 3: 156-63.
• 18 O’Beirn SF, Given HF, Lavelle SM. Experiences with compiling a diagnostic data base on jaundice presenting to a general hospital. Irish J Med Sci 1987; 156: 93-7.
• 19 Malchow-Møller A, Thomsen C, Hilden J. et al. A decision tree for early differentiation between obstructive jaundice and non-obstructive jaundice. Scand J Gastroenterol 1988; 23: 391-401.
• 20 Segaar RW, Wilson JHP, Habbema JDF. et al. Transferring a diagnostic decision aid for jaundice. Neth J Med 1988; 33: 5-15.
• 21 Miller RA. Medical diagnostic decision support systems - past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994; 1: 8-27.
• 22 Pauker SG, Gorry GA, Kassirer JP, Schwartz WB. Towards the simulation of clinical cognition. Taking a present illness by computer. Am J Med, 1976; 60: 981-96.
• 23 Masarie FE, Miller RA, Myers JD. INTER-NIST-1 properties: representing common sense and good medical practice in a computerized medical knowledge base. Comput Biomed Res, 1985; 18: 458-79.
• 24 Barnett GO, Cimino JJ, Hupp JA, Hoffer EP. DXplain. An evolving diagnostic decision-support system. J Am Med Inform Assoc 1987; 258: 67-74.
• 25 Miller RA, Masarie Jr FE. Use of the Quick Medical Reference (QMR) program as a tool for medical education. Method Inform Med 1989; 28: 340-5.
• 26 Warner Jr HR. Iliad: moving medical decision-making into new frontiers. Method Inform Med 1989; 28: 370-2.
• 27 Molino G, Ballarè M, Aurucci PE. et al. Application of artificial intelligence techniques to a well defined clinical problem: jaundice diagnosis. Int J Biomed Comp 1990; 12: 189-202.
• 28 Euricterus Project Management Group – Objective medical decision-making: clinical database for diagnosis of jaundice (EURICTERUS). In: Advances in Biomedical Engineering. Beneken JEW, and Thévenin V. Eds. Amsterdam: IOS Press; 1993: 35-64.
• 29 Molino G, Molino F, Furia D. et al. Computer-aided diagnosis in jaundice: comparison of the knowledge-based and probabilistic approaches. Method Inform Med 1996; 35: 41-51.
• 30 Quaglini S, Stefanelli M, Barosi G. et al. A performance evaluation of the expert system ANEMIA. Comp Biomed Res 1988; 21: 307-23.
• 31 Hilden J, Habbema JDF, Bjerregaard B. The measurement of performance in probabilistic diagnosis. II. Trustworthiness of the exact values of the diagnostic probabilities. Method Inform Med 1978; 17: 227-37.
• 32 Hilden J, Habbema JDF, Bjerregaard B. The measurement of performance in probabilistic diagnosis. III. Methods based on continuous functions of the diagnostic probabilities. Method Inform Med 1978; 17: 238-46.
• 33 Hanley JA, McNeil BJ. The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 1982; 143: 26-36.