Key words visual perception - radiologists - health care quality - computerized tomography -
stroke - acute
Introduction
One of the essential skills of a radiologist is to identify imaging patterns and their
corresponding pathologies. The expertise of radiologists grows as their professional
experience broadens – not only due to increased specialist knowledge, but also due
to continuously improving proficiency in pattern recognition. While specialist knowledge
can be expanded as a result of clinical practice and appropriate reading matter, the
processes that help to continuously improve such pattern recognition have not yet
been fully understood. The higher speed at which findings are identified by an experienced
radiologist is probably attributable to his familiarity with potential patterns, permitting
a scan to be reviewed much faster. The lack of experience of younger radiologists
frequently leads to less focused approaches in respect to differentiating major from
minor diagnostic findings. Major findings predominantly included vessel occlusions
of brain supplying arteries, one cerebral sinus venous thrombosis and one randomly
depicted pulmonary embolism. Minor diagnostic findings included vascular aberrations,
preexisting postischemic defects or pulmonary findings, such as pulmonary nodules,
infiltrates or a pneumothorax and a struma. The precise extent to which the approach
of experienced versus less experienced radiologists differs remains unclear. Recommendations
on best practices for examining radiographic images as well as for optimizing the
detection of findings could be defined based on such deliberations.
Eye tracking is a method that is already widespread in miscellaneous research fields,
especially economics, and has been used, for example, in studies for analyzing marketing
campaigns [1 ]
[2 ]
[3 ]
[4 ]. The method automatically tracks the movement of the eyeball and position of the
pupil while an image or screen is being viewed. Furthermore, it records the duration
that the eye focuses on the relevant areas of the image, without disturbing the assessor.
This method can therefore be used to determine which areas of an image were viewed
and for how long, as well as which areas were potentially completely ignored [5 ].
To potentially provide younger colleagues with specific training and optimize observation
patterns, awareness of efficient techniques for examining scans is of high importance
[6 ]. Thus, the aim of this study was to record observation patterns of radiologists
when reviewing stroke CTs, depending on their professional experience, and to evaluate
the detection/recognition of findings. In addition, the influence of right or wrong
clinical information on the reader’s focus was observed.
Materials and Methods
Subjects
A total of 12 radiologists, with 6 of them working as residents (1 – 5 years of working
experience) and 6 as radiologic specialists (3 were general radiologic senior physicians,
the other 3 were additionally trained as neuroradiologic specialists), were included
for the analysis of image data sets. The average length of working experience was
8 years.
For evaluation, the radiologic assessors used a regular workstation prepared with
an eye tracker on the bottom of the regular radiologic computer screen. The blinded
CT data sets were presented to the assessors through a study supervisor. These assessors
were asked to analyze images according to their normal working pattern. Meanwhile,
their eye movements were recorded with the eye tracker.
In terms of potential interference to eye tracking records, 4 participants had normal
vision or contact lenses and 8 of them had corrected-to-normal vision via glasses.
Each reader had to analyze 20 blinded and randomly presented stroke CT data sets including
non-enhanced CT (NECT), CT angiography (CTA) and CT perfusion (CTP). Diagnostic CT
imaging examinations were indicated by suspected stroke. In order to simulate comparable
clinical reporting conditions in the best manner, clinical information was given prior
to image analysis and the time span per image analysis was limited to a maximum of
10 minutes. To avoid artifacts during fixation measurements and to precisely measure
the time needed for appraisal, pathologies were recorded by the study supervisor through
a think aloud concept.
Measurement of the diagnostic accuracy
The image diagnoses used as the basis for the treatment were used as the "gold standard"
of the available findings in order to subsequently evaluate how many of these findings
were actually found by the assessors and to compare the diagnostic accuracy of experienced
and less experienced assessors.
Measurement of duration of image reading
The maximum time span permitted for image analysis was restricted to 10 minutes. To
simulate the clinical setting of CT image reading, the gaze estimation methods are
based on real-time image processing videos recording the reader’s fixation. This allows
retrospective analysis of the time needed by the radiologists to read data sets.
Order of evaluation of orientations and reconstructions
Additionally, by analyzing these real-time image processing videos, the order of the
image evaluation preferred by the individual assessors for diagnosis was analyzed
retrospectively.
Regions of interest
Furthermore, video analysis allows for investigation of regions of interest (ROIs)
that were defined manually for all patients by encircling every pathologic area in
every cross-sectional view. For 6 randomly chosen patients, the proportionate time
of focusing the ROIs was measured, as well as the time until the assessor’s first
fixation of an ROI, depending on the examiner’s working experience.
Institutional review board approval was granted to retrospectively analyze anonymized
CT data using eye tracking and to record the data of the radiologic assessor.
Eye tracking device
The utilized device (Tobii X2–60 Eye Tracker, Tobii Technology AB, Danderyd, Sweden)
was based on corneal reflection recorded unobtrusively by a tracker located below
the screen. Prior to the recording, a calibration was performed for each participant
to correlate the fixation area on the screen to the coordinates. The distance between
the participant and the monitor was kept constant at 65 cm by means of a mark on the
floor. Gaze data were recorded for both eyes with a sampling rate of 60 Hz. Eye gaze
coordinates were acquired every 16.7 ms. The fixation areas and durations were determined
using a predefined computerized classification algorithm, working with overlays. Consequently,
heat maps of the assessor’s fixation areas, scan paths and the fixation of dedicated
ROIs were generated. Due to the volumetric imaging, ROIs had to be defined manually
for every cross-sectional view containing a pathology.
MSCT acquisition
The examinations were performed using a 256-slice CT scanner (Somatom® Definition FLASH, Siemens AG, Medical Solutions, Forchheim, Germany). For i. v. contrast
administration (Ultravist® -370, Bayer Schering Pharma, Leverkusen, Germany), according to the local standardized
scanning protocols, 40–60 mL are injected followed by a 25-mL saline bolus chaser,
using a constant injection rate of > 4 mL/s.
Images were obtained at a tube voltage of 120 kV and 200 mAs, using a special dose-modulation
template (CARE dose4DTM, Siemens AG, Medical Solutions, Forchheim, Germany) to reduce
the radiation exposure [7 ]. Data sets were reconstructed at a slice thickness of 1.5 mm with a reconstruction
increment of 0.6 mm. For CTP, dynamic scanning was performed at a 66-second biphasic
cine series using the following standardized parameters: 80 kVp, 200 mA, 1-second
rotation time, 1 image per second for 40 seconds with 9 additional images at a rate
of one every third second, as used in the clinical routine [8 ]. For evaluation, different color-coded maps (cerebral blood volume (CBV), cerebral
blood flow (CBF), mean transit time (MTT), and time-to-peak (TTP)) were generated.
Statistical analysis
Statistical analysis was performed using SAS (Version 9.3 for Windows, SAS Institute
Inc., Cary, NC, USA). Inferential statistics were intended to be exploratory (generating
hypotheses) instead of confirmatory and were interpreted accordingly. Hence, p-values
were interpreted according to Fisher, by assessing the metric weight of evidence against
the respective null hypothesis of no effect. P-values were considered significant
if ≤ 0.05 and highly significant if ≤ 0.01. Standard descriptive statistical analyses
were performed.
Categorical variables are described as absolute and relative frequencies.
Results
For every required case, reconstructions of acquired CT data were technically feasible
and assessable. Neuroradiologic reports of diagnostic findings, which were the basis
for the treatment, were defined as the “gold standard” of pathologic findings. Objects
to be detected are categorized as main findings causing the neurologic symptoms, e. g.
early signs of cerebral ischemia, acute arterial occlusions, dissection of the cervical
arteries, cerebral venous sinus thrombosis or findings suspicious for tumor, and as
secondary findings without relevance with respect of the suspected stroke, including
preexisting cerebral infarctions or preexisting vessel pathologies like aneurysms
or atherosclerosis, cervical pathologies that are recorded in the cervical angiography
of brain-supplying vessels, like struma or in one case a cervical hematoma, or thoracic
pathologies containing pulmonary diseases like pneumothorax, pulmonary infiltrates
or pleural effusions, a cardiomegaly in one patient and subcutaneous emphysema.
Overall and itemized diagnostic accuracy
Overall and itemized diagnostic concordance with consensus with respect to main or
secondary findings and considering the working experience are detailed in [Table 1 ]. Secondary or incidental findings in this context are all findings that have no
connection to the symptoms or primary disease (examples are aneurysms, pleural effusions,
struma nodosa, etc.). Diagnostic concordance varied in relation to professional experience,
while no relevant differences in the detection rates of the main findings as a function
of working experience could be observed (residents detected 203 of 252 pathologies
([Table 1 ]) and senior physicians observed 196 of 249 pathologies ([Table 1 ]; the difference results from the fact that a participant did not evaluate two data
sets). Furthermore, the general radiologic specialists observed 102 of 126 pathologies
and the neuroradiologic specialists detected 94 of 123 findings ([Table 1 ]; again, the difference can be explained by the lack of interpretation of two data
sets by a test person). However, with respect to secondary findings, greater differences
between experienced and less experienced assessors were observed. Of the 282 secondary
findings, the resident physicians noticed 158 (0.56 %; [Table 1 ]), while the senior physicians only noticed 122 of 278 (0.44 %; [Table 1 ]; again, the difference can be explained by the lack of interpretation of two data
sets by a test person) secondary findings. With regard to their professional focus,
general radiologic specialists observed 60 of 141 secondary findings and neuroradiologic
specialists mentioned 62 of 137 secondary pathologies ([Table 1 ]; again, the difference can be explained by the lack of interpretation of two data
sets by a test person).
Table 1
Detection rates.Tab. 1 Detektionsraten.
detection rates in relation to available findings
pathologies identified
a: residents
b: senior physicians
c: general radiologists
d: neuro-radiologists
overall pathologies
0.68
0.60
0.61
0.60
main findings
0.81
0.79
0.81
0.76
secondary findings
0.56
0.44
0.43
0.45
Listed above is the ratio of the findings detected from all available findings (overall
pathologies), additionally subdivided into main and secondary findings. The above
(in b) depicted ratio of the findings detected from all available findings (overall
pathologies) is specified as senior physicians with (d: neuroradiologists) or without
a neuroradiological focus (c: general radiologists).
Overall and itemized duration of image reading
The overall time needed to read all available images showed a highly significant difference
between resident physicians and senior physicians, as shown in [Table 2, ]
[Fig. 1 ]. The average time resident physicians spent for reading the available images was
5:41.5 minutes:seconds, whereas senior physicians averaged 4:12 minutes:seconds (neuroradiologic
specialists needed 4:6.7 minutes:seconds and general radiologists needed an average
time of 4:17.1 minutes:seconds to read the available radiologic images). A more detailed
consideration is presented in the three right-hand columns in [Table 2 ]. However, it reveals no significant differences between the proportionate times
needed in regard to the computer tomographic angiographies with amounts of 58 % of
the overall inspection times by residents and 53 % by senior physicians (52 % for
neuroradiologic specialists and 53 % for general radiologic specialists). By considering
the proportionate times of reading different orientations of the CT angiographies,
transverse image orientations are mostly used by all groups. Nevertheless, additional
viewing of coronal orientations of CT angiographies could be observed slightly more
often in neuroradiologists, compared to less specialized readers ([Table 2 ]).
Table 2
Reading times.Tab. 2 Befundungsdauer.
time to read available images in seconds
average time per data set
standard deviation
reading time in transverse orientation
reading time in sagittal orientation
reading time in coronal orientation
all assessors
297.1
113.2
0.48
0.03
0.04
resident physicians
341.5
109.8
0.51
0.04
0.03
senior physicians
252.0
98.2
0.45
0.02
0.05
general radiologic senior physicians
246.7
98.2
0.42
0.03
0.07
neuroradiologic senior physicians
257.1
98.7
0.48
0.01
0.04
The evaluation in the two columns on the left was based on the average time required
for analyzing a case in seconds by all radiologists and according to the subgroups
of assistant doctors and senior physicians (with and without neuroradiological focus).
The three columns on the right show the percentage of time required to analyze the
different section orientations (transverse, sagittal and coronal) of the total observation
time.
Fig. 1 Average diagnosis duration per case for all assessors. Above shown is the average
total diagnostic time in milliseconds per case for all subjects (1), assistant doctors
(2), senior physicians (3), senior physicians with (4) and without a neuroradiological
focus (5). A significant difference (marked by ****) between assistant and senior
physicians (p < 0.0001) is shown, with both values being within the standard deviation
of the respective other value. There was no significant difference between the subgroups
of senior physicians.Abb. 1 Durchschnittliche Diagnosedauer pro Fall für alle Befunder. Oben dargestellt ist
die durchschnittliche Gesamtdiagnostikzeit in Millisekunden pro Fall für alle Probanden
(1), Assistenzärzte (2), Oberärzte (3), Oberärzte mit (4) und ohne neuroradiologischen
Schwerpunkt (5). Ein signifikanter Unterschied (markiert mit ****) zwischen Assistenzärzten
und Oberärzten (p < 0,0001) wird angezeigt, wobei beide Werte innerhalb der Standardabweichung
des jeweils anderen Wertes liegen. Es gab keinen signifikanten Unterschied zwischen
den Untergruppen der Oberärzte.
Order of evaluation of orientations and reconstructions
By means of “screen recordings”, the order of evaluating the images the individual
assessors preferred for their image reporting could retrospectively be analyzed. Mostly,
probands tended to start reporting by inspecting the brain in NECT before altering
to the bone window setting, the CT perfusion or CT angiography in different orientations.
Less experienced assessors more often changed between the different reconstructions
and image series with an average number of alternations of 8.1 (43.1 in relation to
total inspection time), compared to 6.2 average alternations of senior physicians,
resulting in 40.6 in relation to their total inspection time (6.1 with and 6.4 without
neuroradiologic specialization).
Regions of interest
The final investigation included the separate evaluation of the main pathologies in
6 cases. The observed pathologies included a thrombosis of the basilar artery, an
occlusion of the internal carotid artery, a double-sided occlusion of the MCA, a dissection
of the aorta, an occlusion of ipsilateral ICA and MCA and an occlusion of vertebral
and basilar artery. For all patients, regions of interest (ROIs) were defined manually
by encircling the area around the pathology in every cross-sectional view containing
it ([Fig. 2 ]). For recording and evaluation of the recorded data, the Studio Capture software
(Tobii Technology) was used. The proportionate time of focusing the ROIs was measured
and a quotient to the total inspection time was calculated ([Table 3 ]). Furthermore, we evaluated the duration until the reader’s first fixation of the
ROI, depending on the working experience ([Table 4 ]). While no significant differences of the relative time period to focus the region
of interest between the individual groups based on experience could be observed, less
time is needed regarding the time duration until first focusing of the ROI particularly
by neuroradiologic specialists to detect the predominantly neurologic pathologies.
Neuroradiologic specialists detected the pathology most rapidly (average 1:28.2 minutes:seconds;
[Table 4 ]). However, general radiologic specialists on average required 1:54.7 minutes:seconds
and residents 1:54.8 minutes:seconds until first fixation of the ROI (54). Nonetheless,
these results do not achieve the predefined level of significance (p = 0.06).
Fig. 2 ROI labeling. In the present CT angiography scan, a thrombus within the lumen of
the basilar artery can be seen (upper row); in the lower row this pathology is labelled
with an ROI in all 3 planes (examplarily in one slice each, whereby this labeling
must be carried out in all slices of all planes in which the pathology can be seen)
in order to analyze visual fixation of the ROI.Abb. 2 Beschriftung ROI. In der vorliegenden CT-Angiografie ist ein Thrombus im Lumen der
Basilar-Arterie zu sehen (obere Reihe); in der unteren Reihe ist diese Pathologie
in allen 3 Ebenen mit einer ROI markiert (beispielhaft in einer Schicht, wobei diese
Markierung in allen Schichten aller Ebenen, in denen die Pathologie zu sehen ist,
durchgeführt werden muss), um die visuelle Fixierung des ROI zu analysieren.
Table 3
Duration of the fixation of the ROI.Tab. 3 Dauer der Fixation einer „ROI“.
duration of the fixation of the ROI
ratio of duration of the fixation to total inspection time
all assessors
0.128
resident physicians
0.127
senior physicians
0.128
general radiologic senior physicians
0.138
neuroradiologic senior physicians
0.119
In 6 patients out of the 20 patients whose CT data sets were included, ROI-specific
evaluations were also generated to specifically analyze the duration of the fixation
of the diagnostic findings marked with ROI (in seconds), pro rata to the duration
of the overall inspection time.
Table 4
Time until first fixation of the ROI.Tab. 4 Zeit bis zur ersten Fixierung einer „ROI“.
time until first fixation of the ROI
time until first fixation of the ROI (median in seconds)
all assessors
108.6
resident physicians
114.8
senior physicians
93.9
general radiologic senior physicians
114.7
neuroradiologic senior physicians
88.2
Additionally, in these 6 patients out of the 20 patients whose CT data sets were included,
ROI-specific evaluations were generated in order to specifically analyze the time
until the first fixation of the ROI-labelled findings (in seconds), compared to the
reference time (therefore the average duration of senior physicians in seconds is
set as the reference).
Comparing the time period utilized to inspect the different contrast phases (non-contrast
CT, CT angiography and CT perfusion), the ratios indicate that neuroradiologic specialists
prefer to survey the pathology on non-enhanced CTs (42 % of total time used to survey
the ROI in NECT versus 32 % on CT perfusion and 25 % on CT images; [Table 5 ]), whereas the residents, like general radiologists, prefer the observation of the
pathologies on CTA images (general radiologists prefer the observation on CTA with
51 % of total time versus 30 % on CT perfusion and 19 % on NECT; residents prefer
the survey on CTA with 47 % of total time compared to 21 % on CT perfusion and 32 %
on NECT; [Table 5 ]).
Table 5
Duration of the fixation of the ROI in different contrast administrations.Tab. 5 Dauer der Fixation einer „ROI“ in den verschiedenen Kontrastierungsphasen.
duration of the fixation of the ROI itemized to different contrast administrations
in relation to total inspection time
fixation of the ROI in CT perfusion
fixation of the ROI in non-contrast CT
fixation of the ROI in CT angiography
all assessors
0.26
0.31
0.43
resident physicians
0.21
0.32
0.47
senior physicians
0.31
0.31
0.38
general radiologic senior physicians
0.30
0.19
0.51
neuroradiologic senior physicians
0.32
0.42
0.25
Moreover, in these 6 of the 20 included CT datasets, ROI-specific evaluations were
generated in order to specifically analyze the duration of the ROI-labelled findings
in the different contrast administrations (non-enhanced CT, CT angiography and CT
perfusion) in seconds, related to the overall time needed to inspect each dataset.
Discussion
In the clinical routine, neuroradiologic CT imaging in the case of suspected stroke
using unenhanced brain CT, CT angiography of the supra-aortic vessels and CT perfusion
represents a common method to explore the pathology causing the symptoms. CT angiography
offers detailed information about the vascular system and CT perfusion provides valuable
information regarding the brain’s vascular physiology [9 ]
[10 ]
[11 ]. Consequently, radiologists read hundreds of sections of imaging in the case of
a suspected acute ischemic stroke. Compared to reading a single static radiograph,
requirements differ substantially in terms of understanding the cognitive and perceptual
processes of interpreting volumetric data sets like CT stroke imaging. The visual
search for lesions is a process that has been shown to be mainly acquired rather than
innate and that depends on the working experience.
To evaluate the observations of the assessors, this method has so far been applied
for the radiologic fields of mammography and projection of the chest or the extremities
[12 ]
[13 ]
[14 ]
[15 ]. Only a few studies of multislice images have been conducted to date. Furthermore,
no studies were conducted for more complex scenarios such as multimodal diagnostic
imaging, for example in the event of medical emergencies such as strokes [16 ]
[17 ]
[18 ]
[19 ].
Several theories regarding potential sources of error in the detection of pathologies
have been proposed. For instance, the holistic model for the searching strategy applied
by radiologists when reading mammograms, the theory about the satisfaction of search
errors or the assumption of a global perceptual process [20 ]
[21 ]
[22 ]. Research conducted with eye tracking technology has revealed a connection between
the visual dynamics and cognitive processes.
Whether radiologists are explicitly aware of the details or not, training has taught
them to focus attention more efficiently on items with high probability of interest
[23 ].
Some investigations even differentiated between the 3 different phases in which errors
of image reporting can occur. This includes the search, the recognition and the cognitive
process to identify a lesion [24 ].
Overall and itemized diagnostic accuracy
The study focused on the accuracy of search as one of the major pitfalls, revealing
an overall detection rate between 76 % and 81 %, with slight differences correlated
to the level of education. Especially with regard to the accuracy of the recognition
of secondary findings in the group of senior physicians, it should be mentioned that
the lesions were partially fixed but not mentioned. In the context of this study setting,
a clear distinction cannot be made between missed pathologies and incidental findings
not mentioned due to their lack of clinical relevance.
Overall and itemized times of image reading
Kundel and Nodine described a diagnostic accuracy of 70 % when presenting chest radiographs
to radiologists for 200 msec. According to Kundel et al., the visual search of radiographic
images occurs in two steps: starting with the initial rapid primary “global” response,
taking place during the first 40–200 ms, followed by a second “systemic scan”, allowing
for accurate object recognition using foveal vision [25 ]. Systematic viewing is a technique whereby anatomical areas are inspected in a fixed
order to ensure complete inspection of the images. In order to imitate the clinical
setting of reporting as accurately as possible and to include the assumed “systematic
scan” into our analysis, we determined a period of 10 minutes. This allows a detailed
examination.
Furthermore, we investigated the detection error in greater detail by deconstructing
the assessor’s image analysis into the following time ranges: the time until first
detection of the pathology, the duration of the fixation of the pathology and the
proportionate time of observation in different contrast phases (NECT, CTA and CT perfusion).
To the best of the author’s knowledge, previous studies have not investigated these
aspects of image analysis. Neuroradiologic specialists detected the pathology most
rapidly, which could be explained by their superior experience. However, the relative
time needed to fix the pathology does not reveal significant differences between the
groups. In this case, this could be attributed to the common pathologies, which are
easy to assess, even for less experienced assessors.
The differences between the proportions of analyzing images separated into different
contrast phases could also be explained by the neuroradiologists superior experience,
resulting in less time required to analyze the CTA in relation to the total time of
inspection. However, the lack of experience leads residents to spend proportionally
more time on CTA image analysis, compared to the total time.
Visual search is an integral part of the radiologist's interpretation procedure, but
the precise search requirement varies considerably with individual clinical situations.
Such as the purpose of the examination and the radiologist's prior knowledge about
the patient. In many cases, a free search is required, meaning that the radiologist
needs to search the complete data set for any abnormalities, whereas the clinical
situation may require him to focus on specific anatomical areas [26 ]. Although we specified clinical information and observed fixation of clinically
important regions in reference to the symptoms more rapidly in the group of experienced
readers, the influence of clinical information about the patient was not specifically
part of this study and is to be confirmed in further investigations. Our results reveal
that observer’s sensitivity depends on work experience due to systematic scanning
of important areas combined with knowledge about the course of disease. The areas
of missed pathologies mostly included secondary findings without connection to the
primary disease or a lack of methodical approach of the inspection. Thus, eye tracking
can potentially be used to improve the learning curve of teaching residents for radiologic
inspection by teaching them to follow the eye-gaze patterns of expert radiologists.
A limitation is the use of the final radiological reports to define the major and
minor findings. While major findings are most likely be reported, minor findings might
not be. Additionally, the sample size was restricted both by limited availability
of participants and the time required for the individual evaluations. Furthermore,
general radiologists in our institute are very well trained in cross-sectional imaging
diagnostics of neuroradiological findings, as there is no separation between general
and neuroradiological findings in cross-sectional imaging, which can explain the small
differences in performance between general radiologists and neuroradiologists.
In conclusion, our results suggest that the experience of radiologic physicians improves
the detection of major findings by looking at CT images more systematically with inspection
of anatomical areas in a fixed order, as well as an intentional scan of clinically
important regions (like areas causing identifiable symptoms), taking into account
the individual clinical symptoms.
Visual attention was more focused in the case of the experienced physicians and often
less time was needed.
However, all levels of experience seem to be able to improve their performance by
analyzing CT images systematically.
Conclusion
Our results suggest that the experience of radiologic physicians improves the detection
of major findings by looking at CT images more systematically, meaning inspection
of anatomical areas in a fixed order to ensure full coverage, as well as an intentional
scan of clinically important regions (like areas causing identifiable symptoms), taking
into account the individual clinical symptoms.
Visual attention was more focused in the case of the experienced physicians and often
less time was needed.
Clinical relevance of the study
Understanding how radiologic expertise is developed may allow training regimes and
consequently the performance of all groups of experienced and inexperienced radiologists
can be improved.
The speed and thus efficiency of expression as well as the quality of detection rates
can be improved.
Pitfalls can be avoided with knowledge of the approaches.
