Key words
MR-diffusion/perfusion - neural networks - vascular - staging - diagnostic radiology
Introduction
The basic goal of all contrast agent (CA)-based perfusion measurement methods is to
obtain detailed information about the structure and function of the vascular network
by observing its dynamic response to a defined CA bolus. For this purpose, MR imaging
is an ideal measurement method as both high temporal and spatial resolution may be
achieved using current-generation scanners. If the relationship between contrast agent
concentration and signal intensity is known, the contrast agent concentration can
be calculated for each voxel and timestep. After this has been achieved, tissue models
of different complexity may be fitted to the concentration time curves and the results
used for diagnosis, treatment monitoring, or basic research.
The goal of this review is to explore the potential of novel deep learning-based processing
methods which may be able to capture hitherto unknown perfusion parameters such as
temporal curve shape or spatial enhancement patterns. It may be possible to identify,
similarly to MR fingerprinting in structural MR imaging, perfusion signatures which
carry information about the underlying microvascular architecture. As ample literature
on the technical details of perfusion MRI exists, conventional acquisition and processing
methods are only presented in brief.
In the first section, the influence of the vascular architecture and function on the
CA dynamics are reviewed. This is followed by a brief recapitulation of conventional
T1-weighted and T2*-weighted image acquisition methods and their relative strengths
and weaknesses. In the main section, the potential of deep learning-based perfusion
processing is reviewed and discussed, followed by an overview of current and potential
future clinical applications. This is followed by a look at future research directions.
Background
Vascular networks and blood flow
The architecture and integrity of the tissue microvasculature determine the temporal
and spatial signal dynamics in response to an external CA bolus. It must be emphasized
that, while the signal dynamics are highly dynamic and of large magnitude, the measured
blood flow and diffusion effects are largely static over the measurement duration:
the changes in contrast agent concentration do not represent the return of a perturbed
system to its equilibrium – the CA bolus is merely the measurement vehicle with which
static effects such as blood flow, contrast agent extravasation, and diffusion are
measured. The relevant effects can be largely classified into two categories: flow
effects that take place inside the vascular system and exchange effects taking place
between intra- and extravascular spaces. As physiological microvascular flow is nearly
always laminar and therefore deterministic, the flow of each measured intravascular
proton is theoretically predetermined by the vascular architecture and its flow patterns
[1]. Due to resolution limits, however, the precise morphology and flow velocity of
each capillary segment cannot be known. Recent models suggest [2]
[3] that the capillary network is organized according to common principles, and that
the microarchitecture can be described by a few organizational parameters [4]
[5]. This creates the possibility to simulate realistic microvascular networks relatively
easily, which can in turn be used to explore the influence of these organizational
parameters on the CA dynamics [6]. Concepts of statistical mechanics and graph theory may be of use to explore this
space further [7]. Exchange effects arise due to the permeability of the vessel walls, allowing the
transport of fluid and solutes, including the CA itself, between blood and the extravascular
space. This exchange is driven by concentration gradients, hydrostatic and oncotic
pressure differentials and, partly, active transport. Tumor growth is associated with
neoangiogenesis [8], with the newly developed vasculature being significantly more fragile and permeable
than the physiological vasculature [9]. Imaging permeability effects requires much longer measurement times [10], as the exchange effects are at least two magnitudes slower than directed blood
flow.
Basic measurement principles
For the acquisition of T1-weighted perfusion imaging, called dynamic contrast-enhanced
(DCE) MRI, dynamic MRI measurements using a heavily T1-weighted MR sequence with sufficient
spatial and temporal resolution are necessary. This is possible using either spin
echo or gradient echo-based stimulation schemes, although 3D gradient echo sequences
have become the de-facto standard. An example of a DCE acquisition can be seen in
[Fig. 1B]. Recently, compressed sensing-based sequences have been introduced, allowing shorter
acquisition times and adaptive readout schemes. The absolute contrast agent concentration
can be calculated from the relative change in signal intensity before and after application
of the CA. For this, the absolute T1 time of the tissue must be known beforehand,
usually by applying a quantitative T1 mapping sequence such as a variable flip angle
(VFA) or modified Look-Locker (MOLLI) sequence. The main advantage of T1w imaging-based
perfusion measurements is that the signal intensity-contrast agent concentration relationship
is only weakly influenced by tissue effects, allowing the modeling of permeability
effects in which contrast agent leaves the vasculature.
Fig. 1 Comparison of DCE and DSC MRI. A Raw DSC MRI of the human brain using single-shot gradient echo planar imaging at
time point t1 (left) and t10 (right). B Calculated CBF (left) and CBV (right) maps. C Raw DCE MRI of the human brain using a 3D gradient echo TWIST sequence at time point
t1 (left) and t10 (right). D Fitted Tofts model with parameters ktrans (left), vep (right).
Abb. 1 Vergleich zwischen DCE- und DSC-MRT. A Rohdaten einer DSC-Aufnahme des menschlichen Gehirns mittels einer single-shot echo-planaren
Gradientenechosequenz an Zeitpunkt t1 (links) und t10 (rechts). B. Berechnete CBF- (links) und CBV- (rechts) Karten. C Rohdaten einer DCE-Aufnahme des menschlichen Gehirns mittels einer 3D Gradientenecho-TWIST-Sequenz
an Zeitpunkt t1 (links) und t10 (rechts). D Gefittetes Tofts-Modell mit Parametermaps für ktrans (links) und vep (rechts).
For T2*-weighted perfusion imaging, called dynamic susceptibility-weighted (DSC) MRI,
usually a gradient echo sequence is applied, most commonly echo planar imaging due
to its uniquely high acquisition speed [11], as shown in [Fig. 1A]. As the vessel geometry and susceptibility difference between vessel and parenchyma
have a large influence on signal dephasing speed, it is common practice to saturate
the interstitial space in regions of high permeability by applying a small pre-bolus
of contrast agent before starting the dynamic acquisition. This attenuates the interstitial
signal changes during passage of the main bolus, allowing correct quantification of
cerebral blood flow (CBF) and cerebral blood volume (CBV). Still, additional postprocessing
correction of contrast agent extravasation is highly advantageous [12]. The relative signal intensity of T2*-weighted spin echo and gradient echo sequences
is dependent on the vessel geometry inside the voxel. This effect is used in vessel
size imaging (VSI) [13]
[14] or vessel architecture imaging (VAI) [15] to determine vessel diameters.
Quantitative analysis
The challenge of quantitative perfusion and permeability modelling is fitting complex
nonlinear models to noisy measurement data while having to correctly describe the
aberrant contrast agent dynamics created by pathological tissue. Therefore, care has
to be taken when using regularization algorithms in order to not correct away important
abnormalities. From a mathematical viewpoint, dynamic contrast agent curves are stochastic
time series with overlying noise. Depending on the specific measurement sequence,
the noise distribution of the signal intensity [16] and contrast agent concentration is not necessarily Gaussian. This has important
implications for noise estimation and significance testing.
Perfusion modeling
The basis of perfusion modeling is the indicator-dilution theorem, which connects
the local dynamics of centrally injected contrast agent with the local tissue blood
flow and blood volume [17]
[18]. These parameters can be determined directly from the concentration time curve without
having to assume an underlying tissue model. In order to account for variations in
the central circulation, the bolus curve either of a large perfusing artery or an
interpolated “local” vessel is used as an arterial input function (AIF) [19]. The AIF is usually either manually or automatically selected [20]
[21]. The AIF is deconvolved from the voxel CA concentration time curve, resulting in
the tissue residuum function, from which CBF and CBV can be directly calculated. Examples
of the resulting maps are shown in [Fig. 1B]. While the calculation of conventional parameters like CBF, CBV, time to peak (TTP),
and mean transit time (MTT) is well-established and a mainstay of neurooncological
and neurovascular diagnosis, novel processing methods like wavelet-based analysis
[22]
[23], Bayesian vascular models [24], and control-point interpolation methods [25] may be able to capture pathological changes better. In addition, direct inference
from the tissue residuum function may be able to capture important features which
are missed by calculated single parameters. Changes in the microvascular architecture
impact the transit delay of the CA, leading to changes in the residuum function shape.
Permeability modeling
For modeling of permeability effects, a dynamic tissue model with fixed exchange coefficients
is assumed. Parameter fitting is usually accomplished by either nonlinear least squares
(LS) fitting or Bayesian optimization [26]
[27]. Since the introduction of the Tofts [28], Brix [29] and Patlak models [30], increasingly complex models have been proposed. Examples include the 2-compartment
chemical exchange model (2CXM) [31], the compartmental tissue uptake (CTU) model [32], and the adiabatic approximation to the tissue homogeneity (TH) model [33]. The maps resulting from a Tofts model fit can be seen in [Fig. 1 D]. While research of tissue models for DCE perfusion has enjoyed a constant popularity
among MRI researchers and robust fitting algorithms have been developed, clinical
utilization of the derived parameters remains low. Most diagnostic guidelines still
rely on qualitative descriptors of contrast agent bolus curves.
Deep learning
Deep learning algorithms are essentially complex nonlinear functions which can represent
nearly any underlying distribution provided enough variable input is provided [34]
[35]. This essentially implies that most, if not all, conventional processing steps that
are applied to the raw perfusion data can be learned by a neural network. This raises
the question as to whether it is useful to train a neural network to provide known
perfusion parameters as an output. The ultimate goal of perfusion imaging is to provide
functional information not available from purely morphological imaging and to use
it to improve detection rates, diagnostic accuracy, and outcome prediction. Neural
networks can be trained to directly output this information by training them on data
obtained from medical records or histopathological reports, e. g., tumor grading or
staining density. The prediction of clinical outcome parameters directly based on
raw perfusion data is not an easy task, however, and requires both a large amount
of training data and enough training time to provide sensible results. As an intermediate
step, the output parameters of conventional quantitative processing methods can be
learned instead, essentially teaching a neural network the mathematical transformations
behind the processing pipeline. This is far easier than predicting clinical or outcome
parameters, as the quantitative parameter is usually known for each voxel separately,
and there is a directly (quasi-)deterministic relationship between input and output
variables. Therefore, less training data is required, and training converges faster.
Furthermore, even if no new information is gained compared to conventional algorithms,
implementing the processing in a common neural network may have other advantages,
such as higher processing speed or better interoperability between data acquired by
MR devices from different vendors. The raw data from T1- or T2*-weighted perfusion
imaging take the form of a four-dimensional array – three space dimensions are acquired
for each timestep. Two different classes of neural networks have been employed to
model this spatiotemporal data: convolutional neural networks and recurrent neural
networks.
Convolutional neural networks are derived from fully convolutional networks and incorporate
hidden layers which perform spatial convolution steps, helping them capture complex
relationships at different resolution scales. For perfusion imaging, the 3D images
captured at each timestep are usually assigned as different input channels. This has
the advantage of easily capturing spatial relationships but requiring that the input
data always has the same number of timesteps. In addition, the results are not invariant
under temporal shifts, such as when the acquisition was started earlier or later.
A possible solution is four-dimensional convolutional neural networks with modified
loss functions [36]
[37]. Finally, as convolutional neural networks are commonly used for tissue segmentations,
their incorporation into perfusion processing workflow can help in extracting tissue
parameters [38].
Recurrent neural networks, on the other hand, are natively designed for sequential
input data: there is a one-to-one-relationship between each temporal position in the
input data and a network layer [39]. Each layer, in addition to input and output nodes, consists of several hidden nodes,
with weight matrices shared between different timesteps. This makes them invariant
under time shifts, an important advantage when considering perfusion data. Training
recurrent networks has specific challenges such as vanishing or exploding gradient
problems [39]. Several network architectures were designed to deal with these problems, the most
prominent being long short-term memory (LSTM) nets [40]
[41]. The architecture of LSTM designed to learn CBV values from DSC data is shown in
[Fig. 2]. LSTM networks have shown high promise in modeling a wide range of different sequential
problems in radiology, such as in predicting IDH genotype in gliomas [42], breast lesion classification [43], for segmenting tissue or organs [44]
[45], differentiating the origins of spinal metastases [46], and recently for predicting DCE model parameters [47]. It is also possible to learn the necessary transformations for perfusion modelling
of DSC data, as demonstrated in [Fig. 3] (own work). As can be seen in the right part of [Fig. 3B], the root mean squared error between the predicted and the conventionally calculated
CBV is very small.
Fig. 2 Recurrent neural network architecture for the prediction of CBV from DSC contrast
agent curves with N time points. The network consists of L layers with N LSTM cells,
each with M hidden features, in each layer. For each voxel separately, the concentration
at each time point c(tn) is processed by a separate LSTM cell with hidden state hl,n. The weighting function wl,n is the same inside each layer. The last output of the last layer is given as the
input for a fully connected network (FCN) layer with (M, 1) nodes. The final output
is the trained parameter, in this case the CBV.
Abb. 2 Beispielhafte Architektur eines rekurrenten neuronalen Netzwerks für die Vorhersage
des CBV ausgehend von den Kontrastmittelkurven einer DSC-MRT-Aufnahme mit N Zeitpunkten.
Das Netzwerk besteht aus L Schichten mit je N LSTM-Zellen, jede mit M hidden features.
Die Kontrastmittelkonzentrationen c(tn) zu jedem Zeitpunkt werden für jeden Voxel einzeln von einer eigenen LSTM-Zelle mit
dem hidden state hl,n verarbeitet. Die Gewichtungsfunktion wl,n ist für alle Zellen in einem Layer gleich. Der letzte Output des letzten Layers wird
als Input für ein fully connected network (FCN)-Layer mit (M,1)-Neuronen benützt.
Der Outputparameter dieses FCN ist schlussendlich das CBV pro Voxel.
Fig. 3 Demonstration of the capabilities of an exemplary LSTM network. A Exemplary mean squared error (MSE) loss, evaluated on a validation subset, over the
training epochs, showing rapidly decreasing loss when using the ADAM optimizer for
a learning rate of 1e-6. B Comparison of the conventionally obtained CBV, as obtained by a Tikhonov-regularized
singular value decomposition (TiSVD, left) and the learned CBV LSTMpred (middle) on a test case which was not in the training or validation cohort. The right
image shows the root MSE between the CBV values obtained by the two different methods.
Pseudocolor scale is identical across the methods with a range of [0, 30] ml/min.
Abb. 3 Beispiel der Fähigkeiten eines exemplarischen LSTM-Netzwerks. A Beispielhafte Darstellung des mittleren quadratischen Fehlers (MSE loss) eines Validationsdatensatzes
abhängig von der Trainingsepoche. Der Fehler nimmt bei Benutzung des ADAM-Optimizers
und einer learning rate von 1e-6 rasch ab. B Vergleich der mittels eines konventionellen Tikhonov-stabilisierten SVD-Algorithmus
berechneten CBV, (TiSVD, links) und der gelernten CBV (CBV LSTMpred, mittig) anhand eines Testdatensatzes, der nicht Bestandteil der Trainings- oder
Validierungskohorte war. Das rechte Bild zeigt die Wurzel des mittleren quadratischen
Fehlers (RMSE) zwischen den beiden Methoden. Die Pseudofarbskala ist bei allen Darstellungen
einheitlich [0, 30] ml/min.
Model interpretability and error estimation
In order to compare perfusion parameters from different voxels or measurements, it
is necessary to have an estimation of the parameter errors. This error arises from
two main sources: from intrinsic MRI measurement noise, and from deviations between
real voxel tissue and the chosen tissue model. Only when the error is known is it
possible to correctly assess the magnitude of differences and do significance testing.
There are several methods for error estimation, with the most common model-free being
bootstrapping [48]. This method treats the model algorithm as essentially a black box and considers
the output error after artificially adding noise to the input parameter. For deep
learning, different technique-specific methods have been proposed, such as dropout
methods [49] or neural networks based on Bayesian reasoning [50]
[51].
When using black-box machine learning algorithms, the outcome predictions must be
taken at face value as the algorithm natively does not provide a reason for its prediction.
Recently, the field of neural network interpretability has made significant advances
in providing measures which help with the interpretation of results [52]. Specifically for LSTM networks, gradient-based attribution methods [53] or structure modifications allowing direct variable importance output [54] have been proposed. This can be used to extract associations between contrast agent
curve shape and outcome parameters [42].
Current and future clinical applicability
The quantitative evaluation of DSC MRI, that is, perfusion modeling, currently has
far more clinical applications than permeability modeling using DCE MRI. CBF and CBV
maps are used in the diagnosis of stroke, glioma, head-and-neck tumors, and sometimes
in cardiac imaging. On the other hand, most clinical applications of DCE MRI imaging
rely on purely qualitative or semiquantitative assessment. In the PI-RADS guidelines
for the diagnosis of prostate cancer, only the presence or absence of early enhancement
is scored since the available evidence for pharmacodynamic modeling is deemed insufficient
[55]. Similarly, breast cancer diagnosis using the BI-RADS criteria also uses a classification
of the signal dynamics into one of three curve types according to rise speed and washout
[56]. This does not imply, however, that there is no evidence for the usefulness of these
models, and a large number of small-scale studies exist [57]. Widespread adoption of quantitative DCE is hindered by several factors such as
insufficient standardization of acquisition and processing. The relatively poor current
performance of quantitative DCE models in distinguishing healthy from malignant tissue
may stem from deficits in the handling of the noise levels inherent in fast T1w imaging
and in cleanly separating perfusion effects such as bolus delay and dispersion from
permeability effects. This may change in the future, however, as deep learning-based
modelling becomes commonplace. The optimal mathematical framework for correctly handling
both intrinsic and extrinsic noise is given by stochastic analysis, in particular
by stochastic differential equations.
OUTLOOK
Currently, CA-based perfusion MRI is a well-established functional imaging method
with a multitude of appropriate processing methods. While new processing methods continue
to be developed, progress has declined somewhat in recent years. Machine learning
in general, and deep learning in particular, comprise promising new avenues for better
and more reliable processing methods. The inherently stochastic nature of neural networks
represents an ideal fit for modeling the inherently noisy CA dynamics. Due to their
flexibility, these methods may be capable of modelling the complex dynamics inherent
in aberrant blood flow patterns without having to specify a particular model beforehand.
The main challenge in applying deep learning algorithms to perfusion MRI data remains
the necessity of high-quality and plentiful data for training. Not only does the raw
perfusion data need to be acquired under standardized conditions using comparable
sequence parameters, but the trained goal parameters, whether segmentations, clinical
outcome parameters, or conventional perfusion parameters, need to be high-quality
too. Particularly the prediction of clinical outcome parameters is demanding due to
the often highly nonlinear and indirect relationship between perfusion data and final
outcome.
A possible avenue to circumvent the problem of always having to learn the complete
problem set is using physics-informed neural networks, which can learn complex tasks
while preserving physical or heuristic relationships specified as differential equations
[58]. These may be able to directly learn, for example, DCE parameters for a specified
model. An even newer generalization of physics-informed neural networks, universal
neural differential equations, provide an explicit way of doing this [59]
[60]. The disadvantage, however, is that the explicit model-free nature of deep learning
is partially lost.
In conclusion, deep learning-based processing of perfusion MRI data holds high promise
for diagnosis and treatment monitoring in oncology. The novel methods may be uniquely
suited for the inherently noisy time series obtained for each voxel and can learn
almost any sensible parameter. A special focus should be on connecting architectural
modeling and perfusion parameters, as this may allow monitoring of microstructural
changes in the microvascular architecture induced by neoangiogenesis or as treatment
response. Due to the rapid progress in the field, further research is urgently needed.
