Degenerative Adversarial NeuroImage Nets:
Generating Images that Mimic Disease Progression
Simulating images representative of neurodegenerative diseases is important for predicting patient outcomes and for validation of computational models of disease progression. This capability is valuable for secondary prevention clinical trials where outcomes and screening criteria involve neuroimaging. Traditional computational methods are limited by imposing a parametric model for atrophy and are extremely resource-demanding. Recent advances in deep learning have yielded data-driven models for longitudinal studies (e.g., face ageing) that are capable of generating synthetic images in real-time. Similar solutions can be used to model trajectories of atrophy in the brain, although new challenges need to be addressed to ensure accurate disease progression modelling. Here we propose Degenerative Adversarial NeuroImage Net (DaniNet) — a new deep learning approach that learns to emulate the effect of neurodegeneration on MRI \colorblackby simulating atrophy as a function of ages, and disease progression. DaniNet uses an underlying set of Support Vector Regressors (SVRs) trained to capture the patterns of regional intensity changes that accompany disease progression. DaniNet produces whole output images, consisting of 2D-MRI slices that are constrained to match regional predictions from the SVRs. DaniNet is also able to maintain the unique brain morphology of individuals. Adversarial training ensures realistic brain images and smooth temporal progression. We train our model using 9652 T1-weighted (longitudinal) MRI extracted from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. We perform quantitative and qualitative evaluations on a separate test set of 1283 images (also from ADNI) demonstrating the ability of DaniNet to produce accurate and convincing synthetic images that emulate disease progression.
Neurodegenerative diseases are strongly age-related, and the incidence of these diseases is expected to rise in the next few years due to increasing life expectancy. Creating tools that assist our limited understanding of neurodegeneration can help to answer some of the open questions in this field, e.g., What are the causes? How can we improve subtype classification?
Disease progression modelling  analyzes clinical and image-based biomarkers to map out longitudinal changes during chronic diseases. However, modelling temporal neurodegeneration on full resolution MRI is still a major challenge. To tackle this problem, a few simulators have been proposed in the literature [3, 4, 8, 6]. The recent SimulAtrophy  uses a computational model based on fluid mechanics. This approach combines two deformation fields: one from a biophysical model and the other obtained by non-rigid registration of two real images. For a given baseline MRI, the combined deformation field is used to impose the desired level of atrophy and generate the simulated image. SimulAtrophy is extremely resource-demanding and is not scalable to high-resolution images. Exploiting the success of deep learning, Bowles et al.  proposed a framework based on Generative Adversarial Networks (GANs) to model and manipulate MRI directly through the technique of image arithmetic. Although the system is able to introduce or remove atrophy patterns from regions in the brain, it presents a few important limitations: disease progression is modelled linearly and morphological changes are the same across all patients. A more advanced deep learning framework developed to predict face ageing was proposed in . Our idea is to use a similar framework to generate realistic images that also preserve biological/anatomical constraints associated with disease progression.
In this paper we present a new framework that overcomes the limitations of existing approaches. \colorblackFirstly, similar to  we employ adversarial training to ensure a high level of realism in the synthetic MRI. Secondly, we avoid imposing predefined atrophy patterns, prefering to implement novel biological constraints that model neurodegeneration. These constraints control the MRI intensity in each brain region through a set of localised SVRs learnt directly from the data. Finally, we design DaniNet to handle non-imaging characteristics, which we employ here to condition upon age and diagnosis. Experiments evaluate the performance of the various components of DaniNet and demonstrate the ability to produce accurate and realistic synthetic images that emulate disease progression.
In this section we present the main blocks which compose DaniNet. A colour-coded work-flow of our system is depicted in Fig. 1.
The first pre-processing block (shown in blue in Fig. 1) extracts a normalized slice from the MRI . The second component (shown in grey) is a Conditional Deep Autoencoder (CDA) composed of two deep neural networks: an encoder that embeds in a latent space , and a generator that projects the vectors produced by back to the original manifold. Before this projection, the latent vector is conditioned with two variables: i) — a numerical representation \colorblack [0-3] of diagnosis (i.e. cognitively normal, subjective memory concern, early/late mild cognitive impairment, Alzheimer’s disease); and ii) — an index describing age, binned into \colorblack=10 groups. This age discretisation is important for computing deformation loss (block in yellow) to learn morphological changes along the progression. The third component (shown in green), consists of two discriminator networks: i) that drives to produce with a uniform prior and smooth temporal progression; and ii) that drives to produce realistic brain neuroimages. Finally, in the blocks coded in orange, we present the proposed biological constraints used to model disease progression. In the next sub-sections, we will describe each of these components in detail.
The pre-processing block removes irrelevant variation on the data and ensures that intensity values in each voxel only decrease. In an ideal scenario, intensity of T1-weighted MRI decreases with age since tissue density (having high intensity) will reduce, while water content (having low intensity) will increase. However, due to scanner variability and other sources of noise, this may not be the case in practice. To handle these variabilities, each input image is normalised using the following pre-processing steps: i) linear co-registration to a 1mm isotropic MNI template \colorblackusing FLIRT-FSL, ii) skull-stripping using BET-FSL, and iii) intensity normalisation to zero mean and unit standard deviation. Images where pre-processing failed were not included in the training set.
2.2 Conditional Deep Autoencoder
A CDA is an extension of the Deep Autoencoder (DA) with the ability to integrate conditional variables inside the generator . The advantage of training a single end-to-end CDA over training separate DAs is that it avoids overfitting individual groups when longitudinal data are missing. For the architecture of and we follow . The output of the encoder is a feature vector that preserves the brain morphology of the current participant. The output of are synthetic images defined as . The aim of is to learn the mapping between the linear transition of the latent vector conditioned on and and the non-linear transition in the original manifold.
2.3 Adversarial Training
Again following , DaniNet includes two discriminator networks that are trained adversarially with the CDA. The first discriminator guides to generate with a uniform distribution to ensure temporal smoothness, as demonstrated in . Specifically, is trained to distinguish the vector generated by the encoder from samples extracted from . On the other side, is trained in a zero-sum game with the purpose to fool . The objective function used for this adversarial training is
where is the expectation operator, is a vector sampled from , estimates the probability that a vector comes from , and is the latent vector obtained from .
The second discriminator guides the generator to produce realistic brain images. This discriminator is trained to distinguish the synthetic images from the real images in the training set via the following objective function
where estimates the probability that a slice is extracted from a real MRI.
2.4 Biological Constraints
To capture the patterns of image intensity changes that accompany disease progression, our framework uses two separate loss functions during training. These loss functions impose biological constraints that mimic neurodegeneration by ensuring decreased intensity (tissue density) that is consistent with disease progression.
The first loss function operates at the voxel level. Given a synthetic output in age group , we impose that corresponding voxels of each preceding image with have higher-intensity values. Likewise, later images with have lower-intensity values. We express this constraint as
where and are the number of rows and columns in each slice\colorblack, and sgn is the sign function.
The second loss function operates at a regional level to improve spatial consistency across neighbouring voxels. We pre-train a set of SVRs to learn how to predict the rate of intensity progression in fixed, overlapping, data-driven regions mask obtained from hierarchical clustering of local intensity of the training images. On the considered axial slice the clustering algorithm detects 128 regions with an average size of 233 voxels. We train one SVR per region, where the input features are the age at baseline, the age at the follow-up, and diagnosis . We restrict the SVR training to monotonically decreasing data by removing time-points where regional intensity increases. The loss function used to constrain the image sequence (with ) through the regional SVR predictions is
where is the number of regions, is the i-th region-mask, is the prediction of intensity rate progression for the region , and avoids division by 0.
The pre-trained SVRs supply a set of constraints on the generator’s outputs for ensuring accurate emulation of neurodegeneration. The process used to compute the rate of regional intensity progression is summarized in Fig. 2.
2.5 Deformation Loss
Our final loss function ensures consistency with the individual subject over time by minimising the difference between individual input image sequences and the corresponding output sequences. For each input image , the difference is minimised between and a weighted average of two outputs from the nearest age bins:
where reflects the distance between input age and the group and .
In summary, the loss functions and ensure monotonic intensity change to mimic neurodegeneration, and together with ensure realistic brain morphology.
2.6 Training Details, Parameters and Evaluation
Data used in the preparation of this article were obtained from the ADNI database (adni.loni.usc.edu). The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD).
Training data (\colorblackMRI-slices: 9852; participants: 876) and testing (MRI-slices: 1283; participants: 179) are pre-processed T1-weighted MRI from the ADNI dataset. This includes data from multiple sites, scanners, and pre-processing pipelines. We included them all to improve robustness to such variability, and hence increase generalizability. The participants were aged between 63 and 87 years old, with the following distribution of clinical disease stage: 28% cognitively normal, 4% subjective memory concern, 54% mild cognitive impairment and 14% Alzheimer’s disease. Each participant has on average 4.7 MRI spanning 3 years. We select participants in the test set having at least one follow-up visit two years after baseline, to allow sufficient time for observable neurodegeneration to occur.
black The architectures of each network , , , are based on the implementation proposed in . The size of the latent space is fixed to 200. The network’s parameters of and are trained to minimize a total loss defined as
where is the cross entropy obtained by the discriminator on the generated latent vectors, and is the cross entropy obtained by the disctiminator on synthetic images . To train our system we use the stochastic gradient descent solver, ADAM ( = 0.0002, 1 = 0.5) on an NVIDIA GTX TITAN-X GPU card. The training procedure converged after 3000 epochs of random mini-batch of 100 slices with a size of 128128. We also align each test individual to the model by baseline age and diagnosis, then personalise DaniNet with a transfer learning step ( component, see results section 3) that involves an additional 200 training iterations on the single baseline image. Personalisation is essential to tune the model for the specific morphology of the individual’s brain, ready to simulate realistic, personalised followup MRI. To note, only a single MRI from each test subject’s first visit is used to personalise the trained model.
In our evaluation, input are from baseline visits. Follow-up data is the ground truth against which we evaluate generated images using complementary quantitative and qualitative analysis. The quantitative analysis involves the use of the Structural Similarity Index matrix (SSIM). Since this matrix is not able to quantify the actual realism of the obtained images we also perform a complementary qualitative analysis via a survey of clinicians and medical imaging experts that evaluate the synthetic images based on their perception.
|0.43 0.07||0.44 0.06||0.59 0.14||0.60 0.14||0.44 0.08||0.48 0.08||0.61 0.15||0.62 0.16|
Our first experiment is designed to show that DaniNet improves similarity between generated image and ground truth, with respect to the baseline approach developed for face ageing . Specifically, we evaluate the contribution obtained by three components of DaniNet: i) obtained exploiting and , ii) obtained conditioning the system with the clinical diagnosis, and iii) obtained when the learned model is transferred before testing on new data.
For this purpose, the SSIM results obtained using different compositions of , , and are reported in Table 1. Clearly, the best results are obtained when the framework includes all three components. The baseline model produces inferior SSIM. Transfer learning is the feature that provides the highest contribution. A moderate contribution is provided by and . These improvements were assessed statistically using a paired t-test and all p-values were less than 0.0001. Visual results for the configurations -, -- and baseline  are shown in Fig. 3. The error maps, shown in the last row of the figure, confirm our findings. A full simulation with the optimal configuration -- is shown in Fig. 4 where neurodegeneration is apparent in the progression (left to right), most obviously in ventricular expansion and cortical thinning.
Our second experiment tests the qualitative performance of DaniNet based on human visual perception. In a survey, we asked 25 medical imaging experts (neurologists and computer scientists) to evaluate on average 36 sets of three images each, randomly selected from the test set, generated using two different configurations of DaniNet (- and --) and the baseline . These were shown to the user in a random order. The input and the related follow-up were also displayed on the screen as references for the participants. For each set of images, the user was asked to select the closest synthetic image to . Results from the survey confirm that DaniNet is a considerable improvement of the baseline approach. The medical imaging experts selected the configuration -- (mean std) times (), the configuration - times (), and the baseline times (). For only outputs () the users were not happy with any of the generated synthetic images.
4 Conclusion and Future Work
We have proposed and evaluated (quantitatively and qualitatively) a novel deep-learning framework that is able to learn how to emulate the effect of neurodegenerative disease progression on structural MRI. \colorblack The framework produces personalised, realistic output images through a combination of biological constraints, transfer learning, and conditioning upon both fixed and variable non-imaging characteristics. To the best of our knowledge, we are the first to propose a simulator that imitates realistic neurodegeneration by imposing biological constraints. In future work we will extend the framework to simulate entire 3D-MRI with the aid of low-memory techniques . Additionally, focal brain pathologies, such as glioma and white matter hyperintensities, are not currently modelled by our framework; future work will extend to consider such effects. Finally, although we have demonstrated our framework for modelling atrophy in MRI, we believe that this solution can be used with different image modalities (e.g. PET, CT, etc.) and to model disease progression on other organs (e.g lung, retina, etc.).
The authors would like to thank NVIDIA Corporation for the donation of the Titan Xp GPU used for this research.
Data collection and sharing for this project was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 666992.
EPSRC grant EP/M020533/1 supports DCA’s work on this topic. The NIHR UCLH Biomedical Research Centre also supports this work.
-  Blumberg, S. B. , et al.: Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images, In: MICCAI. pp. 118-125. Springer (2018)
-  Bowles, C., et al.: Modelling the progression of Alzheimer’s disease in MRI using generative adversarial networks. In: Medical Imaging 2018: Image Processing. vol. 10574, p. 105741K. International Society for Optics and Photonics (2018)
-  Camara, O., et al.: Phenomenological model of diffuse global and regional atrophy using finite-element methods. TMI 25(11), 1417–1430 (2006)
-  Karaçali, B., et al.: Simulation of tissue atrophy using a topology preserving transformation model. TMI 25(5), 649–652 (2006)
-  Khanal, B., et al.: Simulating longitudinal brain MRIs with known volume changes and realistic variations in image intensity. Frontiers in neuroscience 11, 132 (2017)
-  Modat, M., et al.: Simulating neurodegeneration through longitudinal population analysis of structural and diffusion weighted MRI data. In: MICCAI. pp. 57–64. Springer (2014)
-  Oxtoby, N.P., Alexander, D.C.: Imaging plus x: multimodal models of neurodegenerative disease. Current opinion in neurology 30(4), 371 (2017)
-  Sharma, S., et al.: Evaluation of brain atrophy estimation algorithms using simulated ground-truth data. Medical image analysis 14(3), 373–389 (2010)
-  Zhang, Z., et al.: Age progression/regression by conditional adversarial autoencoder. In: CVPR. pp. 5810–5818 (2017)