Generative adversarial network for segmentation of motion affected neonatal brain MRI
Abstract††Accepted in Medical Image Computing and Computer Assisted Intervention 2019
Automatic neonatal brain tissue segmentation in preterm born infants is a prerequisite for evaluation of brain development. However, automatic segmentation is often hampered by motion artifacts caused by infant head movements during image acquisition. Methods have been developed to remove or minimize these artifacts during image reconstruction using frequency domain data. However, frequency domain data might not always be available. Hence, in this study we propose a method for removing motion artifacts from the already reconstructed MR scans. The method employs a generative adversarial network trained with a cycle consistency loss to transform slices affected by motion into slices without motion artifacts, and vice versa. In the experiments 40 T2-weighted coronal MR scans of preterm born infants imaged at 30 weeks postmenstrual age were used. All images contained slices affected by motion artifacts hampering automatic tissue segmentation. To evaluate whether correction allows more accurate image segmentation, the images were segmented into 8 tissue classes: cerebellum, myelinated white matter, basal ganglia and thalami, ventricular cerebrospinal fluid, white matter, brain stem, cortical gray matter, and extracerebral cerebrospinal fluid. Images corrected for motion and corresponding segmentations were qualitatively evaluated using 5-point Likert scale. Before the correction of motion artifacts, median image quality and quality of corresponding automatic segmentations were assigned grade 2 (poor) and 3 (moderate), respectively. After correction of motion artifacts, both improved to grades 3 and 4, respectively. The results indicate that correction of motion artifacts in the image space using the proposed approach allows accurate segmentation of brain tissue classes in slices affected by motion artifacts.
Keywords:motion correction convolutional neural network cycleGAN neonatal MRI
Important brain development occurs in the last trimester of pregnancy including brain growth, myelination, and cortical gyrification . Magnetic resonance imaging (MRI) is widely used to non-invasively assess and monitor brain development in preterm infants. In spite of ability of MRI to visualize the neonatal brain, motion artifacts caused by the head movement lead to blurry image slices or slices with stripes (see Figure 1). These artifacts hamper image interpretation as well as brain tissue segmentation.
To enable the analysis of images affected by motion artifacts, most studies perform the correction in the frequency domain (k-space) prior to analysis [1, 3]. However, frequency domain data is typically not stored and hence, not available after image reconstruction. Recently, Duffy at al.  and Paware et al.  proposed to use convolutional neural networks (CNNs) to correct motion-corrupted MRI from already reconstructed scans. CNNs were trained to reconstruct simulated motion artifacts that were modelled with a predefined formula. This enforces the network towards an assumed distribution of artifacts. However, in practice, it is difficult to estimate the real distribution of motion. Alternatively, a CNN could be trained to generate images without motion artifacts from images with such artifacts. However, this would require training with paired scans, which are rarely available. To solve this, recently cycleGAN has been proposed to train CNNs for image-to-image transformation with unpaired images .
In this study, we propose to employ a cycleGAN to generate MR slices without motion artifacts from slices affected by motion artifacts in a set of neonatal brain MR scans. The cycleGAN is trained to transform slices affected by motion artifacts into slices without artifacts, and vice versa. To generate slices corrected for motion artifacts, we applied the trained cycleGAN to motion affected slices and we hypothesize that images corrected for motion artifacts allow more accurate (automatic) segmentation. To evaluate this, we use a method exploiting a convolutional neural network to segment scans into eight tissue classes. Moreover, we propose to augment the segmentation training data from the cycleGAN that synthesizes slices with artifacts from slices without the artifacts. We demonstrate that the proposed correction for motion artifacts improves image quality and allows accurate automatic segmentation of brain tissue classes in brain MRI of infants. We also show that the proposed data augmentation further improves segmentation results.
This study includes 80 T2-weighted MRI scans of preterm born infants scanned at average of weeks postmenstrual age (PMA). Images were acquired on a Philips Achieva 3T scanner at University Medical Center Utrecht, the Netherlands. The acquired voxel size was mm and the reconstruction matrix was . The scans were acquired in the coronal plane. In this data set, 60 scans had visible motion artifacts in most of the slices and 20 scans had no visible motion in any slice. The reference segmentation of 10 scans out of 20 scans without motion artifacts were available. The scans were manually segmented into 8 tissue classes: cerebellum (CB), myelinated white matter (mWM), basal ganglia and thalami (BGT), ventricular cerebrospinal fluid (vCSF), white matter (uWM), brain stem (BS), cortical gray matter (cGM), and extracerebral cerebrospinal fluid (eCSF).
Motion artifacts in the neonatal brain MR hamper the diagnostic interpretability and precise automatic segmentation of the brain tissue classes. To address this, we propose to correct motion artifacts in the reconstructed MR scans using a cycleGAN. Thereafter, to evaluate whether the corrected images are suitable for segmentation of brain tissues, a CNN architecture was trained to segment the brain into eight tissue classes. Furthermore, to improve segmentation performance, we proposed to augment the training data by synthesizing images with motion artifacts from the images without artifacts using the cycleGAN.
3.1 Artifact correction network
CycleGAN has been proposed to train image-to-image translation CNNs with unpaired images. Given that obtaining paired scans with and without motion artifacts is difficult, cycleGAN was trained to transform slices affected by motion to slices without motion artifacts and, vice versa (Figure 2). The network architecture consists of two cycles, motion correction and motion generation cycles. The motion correction cycle consists of three networks. Motion correction network () transforms slices affected by motion to slices without motion artifacts. Motion generation network () reconstructs the generated slices without motion artifacts to the original image slices. A discriminator CNN discriminates between generated and real slices without motion artifacts . While the discriminator distinguishes between generated and real slices without motion artifacts, the generator tries to prevent it by generating images which are not distinguishable for the discriminator. Similarly, motion generation cycle transforms slices without motion artifacts to slices affected by motion. The network architecture in both cycles is identical. The generator contains 2 convolution layers with stride of 2, 9 residual blocks , and 2 fractionally strided convolutions with stride proposed in . The discriminator networks have a PatchGAN , which classifies overlapping image patches as fake or real. Two adversial losses  were used in both motion correction network and motion generation network. Furthermore, cycle consistency loss in motion correction network () and motion generation network () were weighted by and were added to adversial losses.
3.2 Segmentation Network
To assess segmentation performance in images affected by motion artifacts, a CNN with Unet-like architecture was trained to segment images into eight tissue classes. The segmentation network consists of a contracting path and an expanding path. The contracting path consists of 10 convolution layers followed by rectified linear units (ReLUs). Every two convolution layers the features were downsampled by max pooling and the feature channels were doubled using the following scheme 32, 64, 128, 256, 512. In the expanding path, an up-sampling is followed by a convolution which halves the number of feature channels. The results are concatenated with the corresponding contracting path and convolved by two convolutional layers followed by a ReLU. In the final layer, one convolutional layer maps each component of the feature vector to the desired number of classes. Batch normalization is applied after all convolutional layers to allow for faster convergence. The network was trained with 3D patches of voxels. The network was trained by minimizing the average of Dice coefficient in all classes between the network output and manual segmentation.
Given that slices affected by motion don’t allow accurate manual annotation, to quantitatively evaluate the proposed method, motion is synthesized in images using the motion generation network. This allows evaluation with the manual annotations performed in images without artifacts. Thereafter, the performance of the segmentation network was evaluated using the Dice coefficient (DC), Hausdorf distance (HD) and mean surface distance (MSD) between manual reference and automatically obtained segmentations. The evaluation was performed in 3D.
To evaluate the proposed method on images with real motion artifacts, the images and the corresponding automatic segmentations before and after motion correction were qualitatively evaluated using 5-points Likert scale. The image quality was scored on a scale from 1 to 5, where 1 indicates uninterpretable images with severe motion artifacts, and 5 indicates excellent image quality. Similarly, automatic segmentations were scored 1 when the segmentation failed, and 5 when the segmentation was very accurate.
5 Experiments and Results
Prior to analysis, the intracranial brain volume was extracted from all scans using Brain Extraction Tool . To train the artifact correction network, 15 scans without motion artifacts and 20 scans with motion artifacts were selected for training. The remaining 5 scans without motion artifacts and 40 scans with motion artifacts were used for testing. From scans without motion artifacts, 700 slices without visible artifacts were selected. Similarly, from the scans with motion artifacts, 714 slices with visible artifacts were selected. The network was trained with a batch size of 4. Adam  was used to minimize the loss function for 100 epochs with a fixed learning rate of 0.00005. was set to 10.
To segment the brain into eight tissue classes, the segmentation network was trained with 5 scans without motion artifacts selected from the 15 training scans used to train the motion correction network. The segmentation network was trained with a batch size of 6. Adam was used to minimize the loss function for 200 epoch and the learning rate was set to 0.0001.
In the experiments, we performed quantitative evaluation of the proposed method through the evaluation of the brain tissue segmentation. First, to determine the upper limit of the segmentation performance, images without artifacts were segmented (Table 1, top row). Second, we aimed to evaluate the segmentation performance in the images with artifacts. However, motion artifacts are prohibitive for accurate manual annotation thus, those were not available for such images. Hence, the motion generation network was used to synthesize images with artifacts from the images without artifacts, for which manual segmentations were available. Segmentation was performed in the synthesized images. (Table 1, second row). Third, using motion correction network, the artifacts were removed from the images with synthesized artifacts and those were subsequently segmented (Table 1, third row). In the previous experiments, the segmentation network was trained only with images without motion artifacts, as only those were manually labelled. However, we hypothesized that the performance would improve when the segmentation would be trained with both types of images. Hence, to obtain images affected by motion that can be used for training, similar to the second experiment, we synthesized training images using motion generation network. In the fourth experiment, we evaluated segmentation network trained with augmented training data, i.e. images with and without motion artifacts on images with synthesized motion artifacts (Table 1, fourth row). Finally, segmentation was performed in images with corrected synthesized artifacts as in the third experiment, and training data for the segmentation was augmented as in the fourth experiment (Table 1, bottom row). The results show that correction of motion artifacts using motion correction network improves the performance (Table 1, second vs. third row). Moreover, results demonstrate that the performance of the segmentation network improves when the training data is augmented (Table 1, second row vs fourth row and third vs. bottom row).
|CBCBCB CBCBCBCB\nprt@next CB||mWM||BGTBGTBGT BGTBGTBGTBGT\nprt@next BGT||vCSF||WMWMWM WMWMWMWM\nprt@next WM||BSBSBS BSBSBSBS\nprt@next BS||cGM||eCSF||Mean|
|Motion Corrected & Augmented||DC||0.91||0.48||0.89||0.84||0.94||0.84||0.80|
vCSFWMWMWM WMWMWMWM\nprt@next WMBSBSBS BSBSBSBS\nprt@next BScGMeCSFMean
To qualitatively evaluate the performance of the motion correction network, 40 scans affected by motion artifacts were corrected using motion correction network. Subsequently, the segmentation network trained with the proposed data augmentation was used to segment the corrected images. Qualitative scoring of the images and segmentations before and after motion correction was performed. The evaluation results show that the median image quality and quality of corresponding automatic segmentations were assigned grade 2 (poor) and 3 (moderate), respectively. After correction of motion artifacts, both improved to grades 3 and 4, respectively. Figure 3 shows examples of images and corresponding segmentations before and after motion correction. This shows that the motion correction network reduces motion artifacts and hence, improves quality of the images and corresponding segmentations. Moreover, the figure shows that our proposed motion augmentation further improves automatic segmentations.
6 Discussion and conclusion
We presented a method for correction of motion artifacts in reconstructed brain MR scans of preterm infants using a cycleGAN. We demonstrate that the proposed artifact correction generates images that are more suitable for (automatic) image segmentation. Additionally, we show that training the segmentation network with the proposed data augmentation further improves segmentation performance.
Unlike previous methods that performed motion correction in the frequency domain (k-space), the proposed method corrects motion artifacts in already reconstructed scans. Given that k-space data is typically not available after scans have been reconstructed and stored, the proposed method allows correction.
To conclude, results demonstrate that correction of motion artifacts in reconstructed neonatal brain MR scans is feasible. Moreover, results show that the proposed motion correction allows automatic brain tissue segmentation in scans affected by motion artifacts. This may improve clinical interpretability and extraction of quantitative markers in images with motion artifacts.
-  Atkinson, D., Hill, D.L., Stoyle, P.N., Summers, P.E., Keevil, S.F.: Automatic correction of motion artifacts in magnetic resonance images using an entropy focus criterion. IEEE Transactions on Medical imaging 16(6), 903–910 (1997)
-  Duffy, B.A., Zhang, W., Tang, H., Zhao, L., Law, M., Toga, A.W., Kim, H.: Retrospective correction of motion artifact affected structural MRI images using deep learning of simulated motion (2018)
-  Godenschweger, F., Kägebein, U., Stucht, D., Yarach, U., Sciarra, A., Yakupov, R., Lüsebrink, F., Schulze, P., Speck, O.: Motion correction in MRI of the brain. Physics in Medicine & Biology 61(5), R32 (2016)
-  Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
-  He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
-  Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
-  Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
-  Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
-  Kostović, I., Jovanov-Milošević, N.: The development of cerebral connections during the first 20–45 weeksâ gestation. In: Seminars in Fetal and Neonatal Medicine. vol. 11, pp. 415–422. Elsevier (2006)
-  Pawar, K., Chen, Z., Shah, N.J., Egan, G.F.: Moconet: Motion correction in 3D MPRAGE images using a convolutional neural network approach. arXiv preprint arXiv:1807.10831 (2018)
-  Smith, S.M.: Fast robust automated brain extraction. Human Brain Mapping 17(3), 143–155 (2002)
-  Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2223–2232 (2017)