Key words
machine learning - hybrid imaging - multiparametric imaging
Background
Machine learning (ML) has become ubiquitous and is considered to be an important technology
for health care data analysis in the future. The scientific interest is high: the
number of publications listed on PubMed with the search query “machine learning” has
increased exponentially from 57 in the year 2000 to 16 722 in 2020. In 2017, > 500
projects received total funding of 264 million USD from the National Institutes of
Health for clinical research projects applying ML techniques [1] and even new journals like Nature Machine Intelligence were founded to pool expertise and build platforms for the increasing number of submitted
papers. Inherently technology-driven, data processing has become an integral element
of radiology since the introduction of a standard to archive and transfer imaging
data in 1985 [2]. For decades, the main focus of imaging research has been improving image quality,
reducing radiation exposure, accelerating image acquisition or – for nuclear medicine
– developing new tracers. Although ML has the potential in multiparametric and hybrid
imaging to accelerate image acquisition [3] or support attenuation correction in positron-emission tomography (PET) [4], a substantial innovation of ML is to support image interpretation, i. e. diagnostic
decision support [5]
[6], across all imaging modalities. Oncologic imaging is a major field of application
for ML. Since cancer is one of the leading causes of death worldwide, treatment options
are rapidly evolving and over the last two decades, health spending on cancer has
increased faster than the increase of cancer incidence (total cost of cancer in Europe
was €199 billion in 2018 [7]). Although new biomarkers like circulating cell-free tumor DNA (i. e., liquid biopsy)
are on the rise [8], imaging plays a crucial role in therapy planning and response assessment in clinical
trials. Despite the broad spectrum of available (semi-)quantitative imaging techniques
in radiology and tracers in nuclear medicine, accepted criteria to define cancer or
evaluate treatment response based on more than one imaging parameter (e. g., the diameter,
RECIST, or glucose consumption, PERCIST) of a tumor are rare (e. g., PIRADS). Multiparametric
oncologic hybrid imaging, i. e., the simultaneous acquisition of anatomical information
and (several) functional tissue parameters using two different scan technologies (PET
combined with computed tomography, PET/CT, or magnetic resonance imaging, PET/MRI)
is aiming to provide deeper insight into tumor biology. Nonetheless, to be superior
to conventional imaging is a challenging task and requires technical knowledge of
all imaging modalities and pitfalls of its combination and an understanding of cancer
biology and the specific mechanism of action of the applied therapy. ML extracts imaging
features and has the potential to support the interpretation and acceptance of multiparametric
oncologic imaging by providing new biomarkers of clinical relevance. In the present
review, we discuss the techniques for image preparation, automated lesion segmentation,
and data analysis. Finally, we will discuss possible approaches to overcome current
limitations.
Machine learning basics
Different types of tasks can be solved using ML, such as image segmentation (e. g.,
delineation of lesions), image classification (e. g., benign versus malignant lesions),
and regression tasks (e. g., estimation of lesion permeability). The main purpose
of ML is to train a mathematical model that, based on the provided data, learns a
representation with respect to the underlying task as shown in [Fig. 1]. The learning model represents any (non-)linear and parametric model that maps the
inputs to the model outputs. The mapping function can be, for example, a neural network
(or any other parametric model) whose parameters can be optimized under some given
cost function. The cost function, also known as the error or loss function, is a quantitative
measure that describes the match between the model predicted output and the desired
target (depending on the type of learning).
Fig. 1 Overview of machine learning (ML) training, validation, and test phase. During training,
the model learns from a pool of (labelled) data – depending on the type of learning
(Fig. 2) – the model parameters. ML hyperparameters are optimized on a separate validation
set. The best trained model is applied during inference/test on new unseen data. The
input data can be images or pre-processed features, with the output (classification
or regression) depending on the underlying application.
The input to the model can be an image, numerical information, or any pre-processed
data of these inputs from previous feature extraction steps, like features as used
in radiomics [9]. The output varies depending on the respective task. While for regression, e. g., image reconstruction or image-to-image translations, a continuous-valued
output per voxel is required, classification tasks, e. g., image segmentation or treatment response prediction, provide a discrete-valued
output on a global (whole image) or local (patch or voxel) scale.
During a training process, ML models are provided evidence in the form of data samples such that the
parameters of the neural network can be learned to predict a reasonable output for
any new (unseen) input. During testing, the model is fixed, and the trained parameters generate the output from new unseen
test data. The models perform a sequence of operations on the inputs to yield the
task-specific output. The main aim of the models is to generalize based on their learned
experience to minimize their associated empirical risk. The models are usually iteratively
trained and tuned on a training and evaluation set, with the final trained model being applied to unseen test data. The training, evaluation, and test sets are disjunct sets with ideally distinct
patients to minimize bias.
Types of learning can be differentiated based on the availability of label information
and the type of label integration during training, as illustrated in [Fig. 2]. In principle, this depends on if and how a human observer is involved during training.
In supervised learning, data samples along with their task-specific labels exist in the database. Labelling
can be very time- and cost-intensive and often requires human interaction, ranging
from data sorting and curating to annotating structures within the image. In semi-supervised learning, both labelled and unlabeled data are included in the learning process. The unlabeled
data provides additional information about the underlying data distribution. Self-supervised
learning circumvents the problem of external labels. The input data itself is used
to guide the learning. In a similar sense, in reinforcement learning the model receives feedback as rewards or penalties, based on its current prediction,
which drives the training procedure. Active learning integrates an oracle into the training procedure which is periodically queried to
either label or select the next most meaningful samples for training. The oracle is
in most cases a human observer but can also be another algorithm. In unsupervised learning, no labelled data is available that could be leveraged to guide the training. The
network purely learns to identify patterns in the data. Common approaches are clustering
[10] (e. g., k-means or Gaussian Mixture Models), principal component analysis (PCA),
(variational) autoencoders [11]
[12], Deep Belief Networks [13], and Generative Adversarial Networks (GANs) [14]. Transfer learning investigates the possibility to transfer knowledge between models or tasks. It can
involve sharing information from simpler to more complex tasks, or from a source domain
to another (but similar) target domain [15]
[16]
[17]. Federated learning
[18]
[19]
[20] trains a model across multiple decentralized devices, where each device holds their
own set of training data and only the model weights are shared across devices. This
allows training across highly heterogeneous datasets. Federated learning has a vast
potential in medicine, where sharing of data across multiple centers is challenging
due to data protection and data privacy [21].
Fig. 2 Overview of types of learning with human observer involvement.
Machine learning-based processing and analysis of multiparametric and hybrid imaging
Machine learning-based processing and analysis of multiparametric and hybrid imaging
In the field of medical imaging, ML methods have been proposed to support the human
observer in examining the task of interest with recent shifting from hand-crafted
radiomic features [22] towards data-driven deep learning features. The applications of ML in the processing
and analysis of multiparametric and hybrid imaging ranges from the acquisition side
to deriving a diagnostic biomarker as depicted in [Fig. 3] as an example.
Fig. 3 Multiparametric and hybrid imaging data processing steps: Acquisition, reconstruction,
post-processing and analysis, with exemplary use cases of machine learning-based methods
within these processing steps.
On the acquisition side, ML can enable acceleration of the imaging sequence – especially
for MRI, in which long protocols can be expected for multiparametric imaging [23]
[24]
[25]. Sampling below the Nyquist-Shannon limit requires incoherent and randomly sampled
data points that can be sparsely represented in a transform domain [26]. Reconstruction is usually performed iteratively with non-linear methods which for
conventional techniques can require substantial computation power and time. ML can
reduce this workload by training an appropriate reconstruction model offline before
usage and which can then allow inference in a few seconds. In some cases, this also
allows further improvement in reconstructed image quality or reduction of acquisition
time (in the order of 2 to 8 times, in some applications for which more degrees of
freedom can be exploited around 16 times or higher) over conventional reconstruction
algorithms [27]
[28]
[29]
[30]
[31].
Similar advancements can be achieved in PET imaging. Instead of administering the
full dose, a reduced dose (in the range of 50 % to 90 % reduced administered dose)
or alternatively shorter imaging duration can be performed with the former being given
preference [32]
[33]. The reduction of long PET reconstruction times in iterative algorithms such as
maximum-likelihood expectation maximization or its incremental updated version, ordered
subset expectation maximization, have been studied in end-to-end trained PET reconstruction
models [34]. To improve the image quality of these reconstructions, imaging data from other
modalities like MRI can be leveraged in a joint reconstruction [35]
[36].
Reliable quantitative PET reconstruction depends on accurate attenuation coefficients
derived from (simultaneous) CT or MR imaging. ML has also shown promising advances
here to predict missing features in images (e. g., bone in MRI) or to learn a more
generalizable realization [4]
[37]. In the case of missing imaging data from other modalities for deriving an attenuation
map, PET data itself can serve as the source with an ML model trained to replicate
CT-derived attenuation maps in an image-to-image translation [38]. Besides attenuation correction, physiological motion can have a severe impact on
the obtained image quality. Motion correction with motion models derived from other
imaging modalities or surrogate measures [39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47] can be integrated [48] to compensate for motion-induced blurring and aliasing in PET.
Before or intertwined with image analysis, segmentation can be employed to focus the
model’s attention on the region of interest, with an additional benefit of automatically
streamlining the processing workflow. These automatic segmentations can support in
segmenting lesions [49]
[50]
[51] or organs/tissues of interest [52]. In this context, multiparametric and hybrid imaging can be utilized to provide
distinct and non-redundant information for better localization of the target region
or to make it more robust with respect to outliers and residual imaging artifacts.
Melanoma lesion segmentation from hybrid data is shown in [Fig. 4] for an ML-based solution in comparison to a manually labelled ground truth.
Fig. 4 Exemplary melanoma lesion segmentation in two patients with a machine learning-based
segmentation network from hybrid imaging data in comparison to the manually labeled
expert ground-truth. The segmented lesions are depicted in red.
The analysis of imaging data for multiparametric and hybrid imaging [53]
[54]
[55] mainly supports in cancer classification (e. g., lung nodules in chest CT [56], skin lesions [57], or lymphoma and lung cancer [58]), disease classification [59]
[60]
[61] and the detection of melanoma [62]
[63]
[64]
[65]
[66], abnormalities, and tumors [67]
[68]
[69]
[70]
[71]
[72]
[73]. Often these models combine contextual, non-imaging, and imaging information in
an end-to-end fashion under the usage of multi-stream convolutional neural networks
(CNNs) to accommodate multiple sources of information (e. g., imaging and non-imaging
data) or representations of the input (e. g., imaging modalities or multiple scales,
orientations) [74]
[75].
The obtained image analysis [76]
[77]
[78] and image-based disease diagnostics and prognostics [79]
[80] can serve as biomarkers that could later be integrated into diagnostic decision-making.
However, despite the high performance, reduced processing time, and improved workflow
demonstrated by ML models, most methods have only been studied in a laboratory environment
and clinical adoption is still limited [81]. So far, ML methods have been primarily proposed with a specific task in mind and
were driven by learning patterns on the provided database. ML is good at finding these
patterns in data but cannot explain how they are connected and to what extent this
estimation is reasonable or reliable. Domain-aware (e. g., population, prevalence,
imaging application, imaging hardware, imaging conditions) and expert knowledge (e. g.,
targeted pathology, extractable information) are valuable information that are only
partially considered within the context of ML-based solutions. Models could however
better adapt to changing scenarios and domains if causal information exchange is considered.
Although multiparametric hybrid data may allow increased information sharing among
imaging data samples, in particular the handling of this multiparametric hybrid data
brings several challenges, like different imaging orientations, modalities, contrasts,
and so on, that need to be addressed and accounted for in ML processing. Furthermore,
widespread usage is also limited by the generalizability of the models due to lack
of widespread and diverse data. Transfer learning or federated learning strategies
could help to mitigate these problems in the future. Transferring knowledge between
domains [82], generalizing models better for different domains [83], including expert’s decision-making into model predictions [84]
[85], or examining the influence of medical imaging meta information [86] can help to shape the next generation of ML models for multiparametric and hybrid
imaging.
Conclusion
In this review, we discussed the current state-of-the-art approaches in ML with a
focus on hybrid imaging. Although the results are promising, there is general skepticism
about ML as a “black box”-like tool. ML can extract data from multiparametric images
and relates this information to biological or clinical endpoints. An important point
of criticism is the missing underlying biological rationale of this entirely data-driven
approach which stands in contrast to biomarker development driven by a biology-based
hypothesis [87]. Of course, radiologic images are influenced by tissue properties (e. g., photon
attenuation, proton density, T1/T2 times, diffusivity, glucose consumption) on a –
more or less – molecular scale, and a validation of imaging features to a histopathologic
or genetic ground truth would increase the acceptance of imaging data analysis. However,
the link between a genetic code or cell surface markers to the Hounsfield unit/signal
intensity/SUV (or the combination) of pixels in a macroscopic CT/MRI/PET image is
rather complex and proving this link might be too ambitious for in-vivo images. This is all the more important as the supposed reference standard, like the
pathology, might not be more precise than the imaging biomarker to be validated [88]. Therefore, a post-hoc generation of hypotheses and a validation through clinical
endpoints might be more preferred for ML techniques. On this note, preference should
be given to self-explaining ML strategies that output their prediction together with
an explanation for that prediction, turning output interpretation into explanation
[89]. To make ML an accepted diagnostic tool, there are several steps ahead. A major
issue for ML is the limitation and the heterogeneity of available data for training.
The Quantitative Imaging Biomarkers Alliance (QIBA, [90]) and The Cancer Imaging Archive (TCIA, [91]) are initiatives aiming to make quantitative imaging more robust and to pool cancer
imaging data. To enable multicenter and multidisciplinary data analysis, the prerequisites
for digital medicine in Europe need to be created. Therefore, the patchwork of regulations
throughout the European health systems including strategies for data security, privacy,
as well as ethical and legal concerns need to be overcome [92]. Besides regulatory and bureaucratic concerns, ML studies need computational power
and engineering effort. Therefore, a digital infrastructure is needed to run ML algorithms
in the clinical routine. Regarding ML systems, a stringent standardization and description
of the analytical methods in publications is crucial. For future implementations in
health care, traceability and auditability of ML systems is required [93]. With these steps, ML can reach its full acceptance and potential in daily clinical
usage.
