Download PDF
Methods Inf Med 1978; 17(03): 157-161
DOI: 10.1055/s-0038-1636435
Original Article
Schattauer GmbH

Use of Receiver Operating Characteristic (ROC) Curves to Evaluate Computer Confidence Threshold and Clinical Performance in the Diagnosis of Appendicitis

DIE VERWENDUNG VON ANNAHME-KENNLINIEN ZUR BEWERTUNG VON COMPUTER-KONFIDENZSCHWELLEN UND KLINISCHEM VORGEHEN BEI DER DIAGNOSE DER APPENDIZITIS
F. T. De Dombal
1   From the University of Leeds, Dept. of Surgery, England
,
Jane C. Horrocks
1   From the University of Leeds, Dept. of Surgery, England
› Author Affiliations
Further Information

Publication History

Publication Date:
16 February 2018 (online)

    Permissions and Reprints

This paper uses simple receiver operating characteristic (ROC) curves (i) to study the effect of varying computer confidence of threshold levels and (ii) to evaluate clinical performance in the diagnosis of acute appendicitis. Over 1300 patients presenting to five centres with abdominal pain of short duration were studied in varying detail. Clinical and computer-aided diagnostic predictions were compared with the »final« diagnosis. From these studies it is concluded the simplistic setting of a 50/50 confidence threshold for the computer program is as »good« as any other. The proximity of a computer-aided system changed clinical behaviour patterns; a higher overall performance level was achieved and clinicians performance levels became associated with the »mildly conservative« end of the computers ROC curve. Prior forecasts of over-confidence or ultra-caution amongst clinicians using the computer-aided system have not been fulfilled.

Die vorliegende Arbeit verwendet Annahme-Kennlinien (Receiver Operation Characteristic Curves) um 1.) den Effekt veränderlicher Computer-Konfidenzschwellen und 2.) das klinische Vorgehen bei der Diagnosestellung der akutön Appendizitis zu beurteilen. Mehr als 1300 Patienten aus fünf medizinischen Zentren mit kurzdauernden Bauchschmerzen wurden mehr oder weniger eingehend untersucht. Klinische und computerunterstützte diagnostische Voraussagen wurden mit der »endgültigen« Diagnose verglichen. Aus den Untersuchungen wird geschlossen, daß eine vereinfachte 50/50 Konfidenz-Schwelle für das Computer-Programm genau so »gut« ist wie jede andere Vorgabe. Bereits das Vorhandensein eines computerunterstützten Systems änderte das klinische Verhaltensmuster; es wurde ein höheres Niveau des Gesamtverhaltens erreicht und die Stufe des klinischen Vorgehens näherte sich dem »leicht konservativen« Ende der Computer-Kennlinie. Voraussagen, daß die ein computerunterstütztes System benutzenden Kliniker entweder allzu vertrauensselig oder übervorsichtig reagieren würden, haben sich nicht bestätigt.

Key-words:

Computer-aided Diagnosis - Appendicitis - ROC Curves

Schlüssel-Wörter:

Compiiterunterstützte Diagnose - Appendizitis - ROC-Kurven (Amiahme-Kennlinien)
 

  • References

  • 1 BJEBBEGAABD B, BRYNITZ S, HOLST-CHRISTENSEN J, KALAJA E, LUND-KBISTENSEN J, HILDEN J, DE DOMBAL F. T, HOBBOCKS Jane C. Computer-aided diagnosis of the acute abdomen; a system from Leeds used on Copenhagen patients. In de Dombal E. T, Grémy F. (Eds) Decision Making and Medical Care: Can Information Science Help?. pp. 165-171. ( Amsterdam: North-Holland; 1976. ).
    Google Scholar
  • 2 CHANG F. C, HOGLE H. H, WELLING D. R. The fate of the negative appendix. Amer. J. Surg 126 ( 1973; ) 752-755.
    CrossrefPubMedGoogle Scholar
  • 3 DE DOMBAL F. T. Surgical diagnosis assisted by a computer. Proc. roy. Soc. B 184 ( 1973; ) 433-440.
    CrossrefPubMedGoogle Scholar
  • 4 DE DOMBAL F. T, HOBBOCKS Jane C. Computer-aided Diagnosis: conclusions from an Overall Experience Involving 4,469 patients. In Anderson J, Forsythe J. M. (Eds) Medinfo 74, Vol. 2. pp. 581-585. ( Amsterdam: North-Holland; 1974. ).
    Google Scholar
  • 5 Editorial: Computers and Surgeons. Lancet. 1977 I. 1241.
    PubMedGoogle Scholar
  • 6 GBUEB R, GUNN A. A, RUXTON A. M. Medical audit in practice. Brit. med. J 1977; I: 957-959.
    PubMedGoogle Scholar
  • 7 GUNN A. A. Computer-aided diagnosis of abdominal pain. J. roy. Coll. Surg. Edinb 21 ( 1976; ) 170-174.
    PubMedGoogle Scholar
  • 8 HOBBOCKS Jane C, MCADAMS W. A. F, DEVBOEDE G, GUNN A. A, ZOLTIE N. Some practical problems in transferring computer-aided diagnostic systems from one geographical area to another. In de Dombal F. T, Grémy F. (Eds) Decision Making and Medical Care: Can Information Science Help ?. pp. 159-164. ( Amsterdam: North-Holland; 1976. ).
    Google Scholar
  • 9 JACOB E. T, BAB-NATHAN N, IUCHTMAN M. Error-rate factor in the management of appendicitis. Lancet. 1975 II. 1032.
    PubMedGoogle Scholar
  • 10 LICHTNER S, PELANZ M. Appendectomy in the Federal Republic of Germany. Med. Care 9 ( 1971; ) 311-330.
    CrossrefPubMedGoogle Scholar
  • 11 LUSTED L. B. Digest 7th International Conference on Medical and Biological Engineering, Stockholm. 1967 Cited by Lusted.
    PubMedGoogle Scholar
  • 12 LUSTED L. B. Clinical decision making. In de Dombal F. T, Grémy F. (Eds) Decision Making and Medical Care: Can Information Science Help ?. pp. 77-98. ( Amsterdam: North-Holland; 1976. ).
    Google Scholar
  • 13 WHITE J. J, SANTILLONA M, HALLER Jr. J. A. Intensive in-hospital observation: a safe way to decrease unnecessary appendicectomy. Amer. Surg 41 ( 1975; ) 793-799.
    PubMedGoogle Scholar
  • 14 WILSON D. H, WILSON P. D, WALMSLET G. L, HOBBOCKS Jane C, DE DOMBAL F. T. Diagnosis of acute abdominal pain in the accident and emergency department. Brit. J. Surg 64 ( 1977; ) 250-254.
    CrossrefPubMedGoogle Scholar
  • 15 WILSON P. D, HOBBOCKS Jane C, YEUNG C. K, LYNDON P. J, PAGE R. E, DE DOMBAL F. T. Simplified computer-aided diagnosis of acute abdominal pain. Brit. med. J 1975; II: 73-78.
    PubMedGoogle Scholar