Collective Intelligence Has Increased Diagnostic Performance Compared with Expert Radiologists in the Evaluation of Knee MRI

Purpose: As widely known, the traditional diagnostic approach that involves only a single physician can result in worrying error rates. Collective intelligence has recently been considered as a possible strategy to pool individual physicians’ diagnoses and reduce misdiagnosis. The goal of this work was to investigate the accuracy of these techniques in a real-world experiment of magnetic resonance (MR) assessment by 13 radiologists.

Methods and Materials: We asked 13 radiologists from a tertiary orthopaedic center to identify the abnormal examinations among 417 knee MR images from the Stanford MRNet Dataset (a low-resolution imaging data set used to train machine-learning predictive models). Radiologists were also asked to assess their confidence about their decisions and the perceived complexity of each case. We assessed each radiologist’s accuracy, in comparison with the accuracy rates obtained by leveraging collective intelligence techniques.

Results: The radiologists’ panel obtained an average accuracy of 0.82. Taking the majority annotation for each examination as the gold standard, we obtained an accuracy of 0.86. Weighing the annotations of the radiologists by their expressed confidence resulted in an accuracy of 0.87; considering only the most accurate raters or selecting the most surprisingly popular annotation, we obtained an accuracy of 0.88 in either situation. All collective intelligence techniques increased diagnostic accuracy on the cases for which the radiologists exhibited a greater misdiagnosis rate, that is, on low-confidence and high-complexity cases.

Conclusion: Collective intelligence techniques were associated with increased diagnostic accuracy compared with the average of the individual radiologists and the most accurate one. These promising results suggest that these techniques merit further consideration for applications in the ground truthing of machine-learning models and, potentially, in clinical settings to reduce diagnostic error.

No conflict of interest has been declared by the author(s).

Publication History

Article published online:
17 December 2020