Generative adversarial network for segmentation of motion affected neonatal brain MRI

Generative adversarial network for segmentation of motion affected neonatal brain MRI

N. Khalili Image Sciences Institute, University Medical Center Utrecht, The Netherlands
   E. Turk Department of Neonatology, Wilhelmina Children’s Hospital, University Medical Center Utrecht, The Netherlands
   M. Zreik Image Sciences Institute, University Medical Center Utrecht, The Netherlands
   M.A. Viergever M.J.N.L. Benders Image Sciences Institute, University Medical Center Utrecht, The Netherlands
Brain Center Rudolf Magnus, University Medical Center Utrecht, The Netherlands
Department of Neonatology, Wilhelmina Children’s Hospital, University Medical Center Utrecht, The Netherlands
Brain Center Rudolf Magnus, University Medical Center Utrecht, The Netherlands
   I. Išgum Image Sciences Institute, University Medical Center Utrecht, The Netherlands
Brain Center Rudolf Magnus, University Medical Center Utrecht, The Netherlands
Abstract
Accepted in Medical Image Computing and Computer Assisted Intervention 2019

Automatic neonatal brain tissue segmentation in preterm born infants is a prerequisite for evaluation of brain development. However, automatic segmentation is often hampered by motion artifacts caused by infant head movements during image acquisition. Methods have been developed to remove or minimize these artifacts during image reconstruction using frequency domain data. However, frequency domain data might not always be available. Hence, in this study we propose a method for removing motion artifacts from the already reconstructed MR scans. The method employs a generative adversarial network trained with a cycle consistency loss to transform slices affected by motion into slices without motion artifacts, and vice versa. In the experiments 40 T2-weighted coronal MR scans of preterm born infants imaged at 30 weeks postmenstrual age were used. All images contained slices affected by motion artifacts hampering automatic tissue segmentation. To evaluate whether correction allows more accurate image segmentation, the images were segmented into 8 tissue classes: cerebellum, myelinated white matter, basal ganglia and thalami, ventricular cerebrospinal fluid, white matter, brain stem, cortical gray matter, and extracerebral cerebrospinal fluid. Images corrected for motion and corresponding segmentations were qualitatively evaluated using 5-point Likert scale. Before the correction of motion artifacts, median image quality and quality of corresponding automatic segmentations were assigned grade 2 (poor) and 3 (moderate), respectively. After correction of motion artifacts, both improved to grades 3 and 4, respectively. The results indicate that correction of motion artifacts in the image space using the proposed approach allows accurate segmentation of brain tissue classes in slices affected by motion artifacts.

Keywords:
motion correction convolutional neural network cycleGAN neonatal MRI

1 Introduction

Important brain development occurs in the last trimester of pregnancy including brain growth, myelination, and cortical gyrification [9]. Magnetic resonance imaging (MRI) is widely used to non-invasively assess and monitor brain development in preterm infants. In spite of ability of MRI to visualize the neonatal brain, motion artifacts caused by the head movement lead to blurry image slices or slices with stripes (see Figure 1). These artifacts hamper image interpretation as well as brain tissue segmentation.

To enable the analysis of images affected by motion artifacts, most studies perform the correction in the frequency domain (k-space) prior to analysis [1, 3]. However, frequency domain data is typically not stored and hence, not available after image reconstruction. Recently, Duffy at al. [2] and Paware et al. [10] proposed to use convolutional neural networks (CNNs) to correct motion-corrupted MRI from already reconstructed scans. CNNs were trained to reconstruct simulated motion artifacts that were modelled with a predefined formula. This enforces the network towards an assumed distribution of artifacts. However, in practice, it is difficult to estimate the real distribution of motion. Alternatively, a CNN could be trained to generate images without motion artifacts from images with such artifacts. However, this would require training with paired scans, which are rarely available. To solve this, recently cycleGAN has been proposed to train CNNs for image-to-image transformation with unpaired images [12].

In this study, we propose to employ a cycleGAN to generate MR slices without motion artifacts from slices affected by motion artifacts in a set of neonatal brain MR scans. The cycleGAN is trained to transform slices affected by motion artifacts into slices without artifacts, and vice versa. To generate slices corrected for motion artifacts, we applied the trained cycleGAN to motion affected slices and we hypothesize that images corrected for motion artifacts allow more accurate (automatic) segmentation. To evaluate this, we use a method exploiting a convolutional neural network to segment scans into eight tissue classes. Moreover, we propose to augment the segmentation training data from the cycleGAN that synthesizes slices with artifacts from slices without the artifacts. We demonstrate that the proposed correction for motion artifacts improves image quality and allows accurate automatic segmentation of brain tissue classes in brain MRI of infants. We also show that the proposed data augmentation further improves segmentation results.

Figure 1: Examples of coronal slices from T2-weighted MRI acquired in preterm born infants at 30 weeks postmenstrual age affected by motion artifacts. Structures outside the neonatal cranium have been masked out.

2 Data

This study includes 80 T2-weighted MRI scans of preterm born infants scanned at average of weeks postmenstrual age (PMA). Images were acquired on a Philips Achieva 3T scanner at University Medical Center Utrecht, the Netherlands. The acquired voxel size was mm and the reconstruction matrix was . The scans were acquired in the coronal plane. In this data set, 60 scans had visible motion artifacts in most of the slices and 20 scans had no visible motion in any slice. The reference segmentation of 10 scans out of 20 scans without motion artifacts were available. The scans were manually segmented into 8 tissue classes: cerebellum (CB), myelinated white matter (mWM), basal ganglia and thalami (BGT), ventricular cerebrospinal fluid (vCSF), white matter (uWM), brain stem (BS), cortical gray matter (cGM), and extracerebral cerebrospinal fluid (eCSF).

3 Method

Motion artifacts in the neonatal brain MR hamper the diagnostic interpretability and precise automatic segmentation of the brain tissue classes. To address this, we propose to correct motion artifacts in the reconstructed MR scans using a cycleGAN. Thereafter, to evaluate whether the corrected images are suitable for segmentation of brain tissues, a CNN architecture was trained to segment the brain into eight tissue classes. Furthermore, to improve segmentation performance, we proposed to augment the training data by synthesizing images with motion artifacts from the images without artifacts using the cycleGAN.

3.1 Artifact correction network

CycleGAN has been proposed to train image-to-image translation CNNs with unpaired images. Given that obtaining paired scans with and without motion artifacts is difficult, cycleGAN was trained to transform slices affected by motion to slices without motion artifacts and, vice versa (Figure 2). The network architecture consists of two cycles, motion correction and motion generation cycles. The motion correction cycle consists of three networks. Motion correction network () transforms slices affected by motion to slices without motion artifacts. Motion generation network () reconstructs the generated slices without motion artifacts to the original image slices. A discriminator CNN discriminates between generated and real slices without motion artifacts . While the discriminator distinguishes between generated and real slices without motion artifacts, the generator tries to prevent it by generating images which are not distinguishable for the discriminator. Similarly, motion generation cycle transforms slices without motion artifacts to slices affected by motion. The network architecture in both cycles is identical. The generator contains 2 convolution layers with stride of 2, 9 residual blocks [5], and 2 fractionally strided convolutions with stride proposed in [7]. The discriminator networks have a PatchGAN [6], which classifies overlapping image patches as fake or real. Two adversial losses [4] were used in both motion correction network and motion generation network. Furthermore, cycle consistency loss in motion correction network () and motion generation network () were weighted by and were added to adversial losses.

Figure 2: The CycleGAN consists of two cycles: motion correction and motion generation. In the motion correction cycle, first network is trained to transform slices affected by motion into slices without motion artifacts (), the second network is trained to transform the generated slices without motion artifacts back to the original slices (), and the third network discriminates between real and synthesized slices without motion artifacts (). In the motion generation network, motion was added to the slices without motion artifacts (), motion correction network transforms generated slices to the original slices (), and the discriminator network discriminates between real and fake slices affected by motion artifacts ()

3.2 Segmentation Network

To assess segmentation performance in images affected by motion artifacts, a CNN with Unet-like architecture was trained to segment images into eight tissue classes. The segmentation network consists of a contracting path and an expanding path. The contracting path consists of 10 convolution layers followed by rectified linear units (ReLUs). Every two convolution layers the features were downsampled by max pooling and the feature channels were doubled using the following scheme 32, 64, 128, 256, 512. In the expanding path, an up-sampling is followed by a convolution which halves the number of feature channels. The results are concatenated with the corresponding contracting path and convolved by two convolutional layers followed by a ReLU. In the final layer, one convolutional layer maps each component of the feature vector to the desired number of classes. Batch normalization is applied after all convolutional layers to allow for faster convergence. The network was trained with 3D patches of voxels. The network was trained by minimizing the average of Dice coefficient in all classes between the network output and manual segmentation.

4 Evaluation

Given that slices affected by motion don’t allow accurate manual annotation, to quantitatively evaluate the proposed method, motion is synthesized in images using the motion generation network. This allows evaluation with the manual annotations performed in images without artifacts. Thereafter, the performance of the segmentation network was evaluated using the Dice coefficient (DC), Hausdorf distance (HD) and mean surface distance (MSD) between manual reference and automatically obtained segmentations. The evaluation was performed in 3D.

To evaluate the proposed method on images with real motion artifacts, the images and the corresponding automatic segmentations before and after motion correction were qualitatively evaluated using 5-points Likert scale. The image quality was scored on a scale from 1 to 5, where 1 indicates uninterpretable images with severe motion artifacts, and 5 indicates excellent image quality. Similarly, automatic segmentations were scored 1 when the segmentation failed, and 5 when the segmentation was very accurate.

5 Experiments and Results

Prior to analysis, the intracranial brain volume was extracted from all scans using Brain Extraction Tool [11]. To train the artifact correction network, 15 scans without motion artifacts and 20 scans with motion artifacts were selected for training. The remaining 5 scans without motion artifacts and 40 scans with motion artifacts were used for testing. From scans without motion artifacts, 700 slices without visible artifacts were selected. Similarly, from the scans with motion artifacts, 714 slices with visible artifacts were selected. The network was trained with a batch size of 4. Adam [8] was used to minimize the loss function for 100 epochs with a fixed learning rate of 0.00005. was set to 10.

To segment the brain into eight tissue classes, the segmentation network was trained with 5 scans without motion artifacts selected from the 15 training scans used to train the motion correction network. The segmentation network was trained with a batch size of 6. Adam was used to minimize the loss function for 200 epoch and the learning rate was set to 0.0001.

In the experiments, we performed quantitative evaluation of the proposed method through the evaluation of the brain tissue segmentation. First, to determine the upper limit of the segmentation performance, images without artifacts were segmented (Table 1, top row). Second, we aimed to evaluate the segmentation performance in the images with artifacts. However, motion artifacts are prohibitive for accurate manual annotation thus, those were not available for such images. Hence, the motion generation network was used to synthesize images with artifacts from the images without artifacts, for which manual segmentations were available. Segmentation was performed in the synthesized images. (Table 1, second row). Third, using motion correction network, the artifacts were removed from the images with synthesized artifacts and those were subsequently segmented (Table 1, third row). In the previous experiments, the segmentation network was trained only with images without motion artifacts, as only those were manually labelled. However, we hypothesized that the performance would improve when the segmentation would be trained with both types of images. Hence, to obtain images affected by motion that can be used for training, similar to the second experiment, we synthesized training images using motion generation network. In the fourth experiment, we evaluated segmentation network trained with augmented training data, i.e. images with and without motion artifacts on images with synthesized motion artifacts (Table 1, fourth row). Finally, segmentation was performed in images with corrected synthesized artifacts as in the third experiment, and training data for the segmentation was augmented as in the fourth experiment (Table 1, bottom row). The results show that correction of motion artifacts using motion correction network improves the performance (Table 1, second vs. third row). Moreover, results demonstrate that the performance of the segmentation network improves when the training data is augmented (Table 1, second row vs fourth row and third vs. bottom row).

CBCBCB CBCBCBCB\nprt@next CB mWM BGTBGTBGT BGTBGTBGTBGT\nprt@next BGT vCSF WMWMWM WMWMWMWM\nprt@next WM BSBSBS BSBSBSBS\nprt@next BS cGM eCSF Mean
Motion Free DC
HD
MSD
Motion Synthesized DC
HD
MSD
Motion Corrected DC  0.68  0.85
HD
MSD  0.41  0.16
Motion Augmented DC
HD 40.19 27.42 30.85 15.03
MSD
Motion Corrected & Augmented DC  0.91  0.48  0.89  0.84  0.94  0.84  0.80
HD 26.83 17.77 14.40  7.63 24.99
MSD  0.45  1.89  0.44  0.29  0.19  0.20  0.51

vCSFWMWMWM WMWMWMWM\nprt@next WMBSBSBS BSBSBSBS\nprt@next BScGMeCSFMean

Table 1: Performance of brain tissue segmentation into eight tissue classes. The evaluation of segmentation was performed 1) on scans without motion artifacts (Motion Free) 2) on the same scans with synthesized motion using motion generation network (Motion Synthesized) 3) on the scans where synthesized motion were corrected using motion correction network (Motion Corrected). The segmentation network was retrained with motion augmented scans using motion generation network. The evaluation of segmentation was performed 4) on the scans with synthesized motion using motion generation network (Motion Augmented) 5) on the scans where synthesized motion were corrected (Motion Corrected and Augmented)

To qualitatively evaluate the performance of the motion correction network, 40 scans affected by motion artifacts were corrected using motion correction network. Subsequently, the segmentation network trained with the proposed data augmentation was used to segment the corrected images. Qualitative scoring of the images and segmentations before and after motion correction was performed. The evaluation results show that the median image quality and quality of corresponding automatic segmentations were assigned grade 2 (poor) and 3 (moderate), respectively. After correction of motion artifacts, both improved to grades 3 and 4, respectively. Figure 3 shows examples of images and corresponding segmentations before and after motion correction. This shows that the motion correction network reduces motion artifacts and hence, improves quality of the images and corresponding segmentations. Moreover, the figure shows that our proposed motion augmentation further improves automatic segmentations.

Figure 3: Examples of slices affected by motion artifacts and the corresponding tissue segmentation in neonatal MRI. 1st column: A motion affected slice; 2nd column: Automatic segmentation when the network was trained on slices without motion artifacts; 3rd column: Automatic segmentation, network trained on slices with augmented motion; 4th column: A motion corrected slice; 5th column: Automatic segmentation result on the corrected slice; 6th column: Automatic segmentation results on the corrected slice when the network was trained with data augmentation.

6 Discussion and conclusion

We presented a method for correction of motion artifacts in reconstructed brain MR scans of preterm infants using a cycleGAN. We demonstrate that the proposed artifact correction generates images that are more suitable for (automatic) image segmentation. Additionally, we show that training the segmentation network with the proposed data augmentation further improves segmentation performance.

Unlike previous methods that performed motion correction in the frequency domain (k-space), the proposed method corrects motion artifacts in already reconstructed scans. Given that k-space data is typically not available after scans have been reconstructed and stored, the proposed method allows correction.

To conclude, results demonstrate that correction of motion artifacts in reconstructed neonatal brain MR scans is feasible. Moreover, results show that the proposed motion correction allows automatic brain tissue segmentation in scans affected by motion artifacts. This may improve clinical interpretability and extraction of quantitative markers in images with motion artifacts.

References

  • [1] Atkinson, D., Hill, D.L., Stoyle, P.N., Summers, P.E., Keevil, S.F.: Automatic correction of motion artifacts in magnetic resonance images using an entropy focus criterion. IEEE Transactions on Medical imaging 16(6), 903–910 (1997)
  • [2] Duffy, B.A., Zhang, W., Tang, H., Zhao, L., Law, M., Toga, A.W., Kim, H.: Retrospective correction of motion artifact affected structural MRI images using deep learning of simulated motion (2018)
  • [3] Godenschweger, F., Kägebein, U., Stucht, D., Yarach, U., Sciarra, A., Yakupov, R., Lüsebrink, F., Schulze, P., Speck, O.: Motion correction in MRI of the brain. Physics in Medicine & Biology 61(5),  R32 (2016)
  • [4] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  • [5] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  • [6] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
  • [7] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
  • [8] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  • [9] Kostović, I., Jovanov-Milošević, N.: The development of cerebral connections during the first 20–45 weeks’ gestation. In: Seminars in Fetal and Neonatal Medicine. vol. 11, pp. 415–422. Elsevier (2006)
  • [10] Pawar, K., Chen, Z., Shah, N.J., Egan, G.F.: Moconet: Motion correction in 3D MPRAGE images using a convolutional neural network approach. arXiv preprint arXiv:1807.10831 (2018)
  • [11] Smith, S.M.: Fast robust automated brain extraction. Human Brain Mapping 17(3), 143–155 (2002)
  • [12] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2223–2232 (2017)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
374734
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description