Towards generative adversarial networks as a new paradigm for radiology education

Towards generative adversarial networks as a new paradigm for radiology education

Samuel G. Finlayson
Department of Systems Biology
Harvard Medical School
Boston, MA 02130
&Hyunkwang Lee*
School of Engineering and Applied Sciences
Harvard University
&Isaac S. Kohane
Department of Biomedical Informatics
Harvard Medical School
Boston, MA 02130
&Luke Oakden-Rayner
School of Public Health
The University of Adelaide
Adelaide, SA 5000
These authors contributed equally.

Medical students and radiology trainees typically view thousands of images in order to ”train their eye” to detect the subtle visual patterns necessary for diagnosis. Nevertheless, infrastructural and legal constraints often make it difficult to access and quickly query an abundance of images with a user-specified feature set. In this paper, we use a conditional generative adversarial network (GAN) to synthesize pixel pelvic radiographs that can be queried with conditioning on fracture status. We demonstrate that the conditional GAN learns features that distinguish fractures from non-fractures by training a convolutional neural network exclusively on images sampled from the GAN and achieving an AUC of on a held-out set of real images. We conduct additional analysis of the images sampled from the GAN and describe ongoing work to validate educational efficacy.


Towards generative adversarial networks as a new paradigm for radiology education

  Samuel G. Finlaysonthanks: These authors contributed equally. Department of Systems Biology Harvard Medical School Boston, MA 02130 Hyunkwang Lee* School of Engineering and Applied Sciences Harvard University Isaac S. Kohane Department of Biomedical Informatics Harvard Medical School Boston, MA 02130 Luke Oakden-Rayner School of Public Health The University of Adelaide Adelaide, SA 5000


noticebox[b]Machine Learning for Health (ML4H) Workshop at NeurIPS 2018.\end@float

1 Introduction

Medical trainees must view thousands of images over the course of their training in order to achieve competency in visual diagnosis (Wang, 2012). Nevertheless, medical images pose unique legal and technical challenges that make it difficult to widely disperse images for training between sites or rapidly and simply query data for training purposes within a site (Ching et al., 2018).

In this paper, we propose medical generative adversarial networks (GANs) as a method for developing (human) training tools in the field of radiology. Utilizing the architecture developed by Karras et al. (2017), we train a conditional GAN over fractured and non-fractured hip radiographs using the dataset utilized in Gale et al. (2017). We sample conditional images at 1K resolution, and perform in silico experiments to validate their capacity to re-capitulate key visual features distinguishing fractures from healthy hips. Finally, we describe human experiments that are under development at time of workshop submission, but will more fully elucidate the comparative benefit of GAN-generated images in training medical professionals.

2 Background

2.1 Generative Adversarial Networks

Generative Adversarial Networks (GANs) are a framework proposed by Goodfellow et al. (2014) to estimate generative models over rich data distributions. The key insight of the GAN framework is to pit a generator model against a discriminator model : The discriminator is optimized to distinguish real images from synthetic images, whereas the generator is optimized to reproduce the true data distribution by generating images that are difficult for the discriminator to differentiate.

Taken together, and can be said to engage in a two-player min-max game with the following objective function


where is a real image from the true data distribution , and is a noise vector sampled from distribution (e.g., uniform or Gaussian distribution).

Conditional GANs  Gauthier (2014); Mirza and Osindero (2014) are an extension of the GAN game, where both the generator and discriminator receive additional conditioning variables , yielding and . This formulation allows to generate images conditioned on variables . Conditional GANs have been used to extend the GAN framework to many applications, including generating images based on specific features such as text descriptions. Karras et al. (2017) proposed a GAN architecture that starts with small images and progressively adds layers to grow them to higher resolutions throughout training.

There has been considerable debate in recent years pertaining to the extent to which GANs are truly ”distribution learners” and their utility for tasks such as data augmentation (Santurkar et al., 2017; Li et al., 2018; Antoniou et al., 2017). This may imply that in the immediate term, some degree of caution should be taken when using GANs in clinical applications.

2.2 Motivation and Related Work

Medical imaging poses a paradoxical pair of challenges from the perspective of information technology: On the one hand, medical images create a big data problem: millions of medical images are ordered each year within the United States alone, each of which must be processed, interpreted by medical experts, and stored. Nevertheless, most who work with medical data suffer from a small data problem: medical images are generally siloed and incompletely annotated, making it difficult in many cases for individual researchers and medical trainees to compile sufficient examples for human or machine learning tasks (Ching et al., 2018).

A number of attempts have been made to apply GANs to medical imaging datasets (Nie et al., 2017; Dar et al., 2018; Wolterink et al., 2017; Hu et al., 2017; Guibas et al., 2017; Korkinof et al., 2018; Beers et al., 2018), including more recently the use of conditional GANs for data augmentation in training medical machine learning classifiers (Frid-Adar et al., 2018; Wu et al., 2018). Seemingly unexplored, however, is the possibility of utilizing GANs as a source of training images for humans.

There are several technical and practical reasons for which clinical education may be an ideal deployment setting for medical GANs. Educational environments are comparatively lower stakes than clinical settings, and do not need connect with EHR or PACS systems. As the data backbone for more advanced educational tools, GANS could provide an exceptionally large set of human training examples with comparatively small and entirely self-contained storage requirements. To boot, GANs could in theory allow for human-driven or programmatic conditioning on clinical attributes, which could be used to build rapid and interactive learning experiences.

Educational tools need not cover the full breadth of a clinical domain to provide utility to a learner; for example, consider the frequently visited webpage ”Normal Chest X-Ray: Train Your Eye”, which consists entirely of 500 examples of normal chest-xrays (see: This could mean that medical GANs could be useful in an educational setting even if they only learn to sample from a small subset of the data distribution.

3 Methods

3.1 Data and Model

We trained a generative adversarial network to generate high resolution () hip images conditioned on fracture status.

Institutional approval was granted for all stages of this project. Data for this study consisted of 11,734 pelvic X-ray images of which 3,289 images contain fractures. Data were obtained from the clinical radiology archive at The Royal Adelaide Hospital as described in Gale et al. (2017). In brief, data were preprocessed using a series of convolutional neural networks to identify and extract the region of the pelvic X-rays that contain the hip. Diagnoses were extracted from radiology reports and curated by a practicing radiologist. Right hips were flipped across the vertical axis so that all hips appeared to be left hips.

To implement our model, we leveraged the code accompanying the original progressive GANs paper (Karras et al., 2017). We conducted several initial experiments (not shown) conditioning on simple features such as anatomic location and with smaller resolutions. Also not shown are additional experiments using a customization of the StackGAN architecture (Zhang et al., 2017), which yielded lower resolution images and worse conditioning. Due to resource constraints, the final GAN was only allowed to train for four days on two NVIDIA Titan GPUs, for a total of 8,800,000 images exposed to the discriminator.

3.2 Experiments

3.2.1 Fracture Classification

To assess whether our conditional GAN was properly conditioning on fractures, we conducted a series of experiments with classification models. To conduct these experiments, we split data our data into training (85%) and test sets (15%) by patient. We then trained our GAN exclusivley on the training set, and generated two additional ”training sets” of the same number and class composition as the real training set by sampling from the conditional GAN with different seed values. For all classification models, Inception-v3 pretrained on ImageNet was fine-tuned after the last fully-connected layers replaced with a global average pooling, a fully-connected layer, and a sigmoid layer.

Our first test was a conditional variant of the boundary distortion test described in Santurkar et al. (2017). To execute this test, we first trained and compared classifiers on the real training set and as well as classifiers exclusively trained on each of the two GAN-sampled training sets. Each model was then evaluated on the test set of images held out from the classifier and GAN training.

To further test the characteristics of the GAN-sampled images, we trained classifiers using each of the GAN datasets as augmentation for the real data and for each other, as well as all datasets together. We additionally created one more baseline model by doubling the effective size of the real training set using traditional data augmentation.

3.2.2 Nearest-Neighbors and t-SNE Analysis

We next conducted experiments to test whether our model was merely memorizing images from the training set. To do so, we adapted the evaluation of Chen and Koltun (2017) and Karras et al. (2017), comparing sampled images to near-neighbors in the training set. To find the nearest-neighbors of generated images, we trained VGG16 on the real images in the training and computing activation embeddings at each intermediate convolution layers. We then identified the nearest neighbor in the training set in L1 distance for each such layer. Finally, we utilized the same VGG model to generate t-SNE maps of 256 randomly sampled images from each class in each training set (Hinton and Roweis, 2003).

Figure 1: Randomly sampled images generated by the GAN (3 from each class using a pre-specified random seed), along with nearest-neighbors in the training set in feature space of a VGG16 model trained on radiographs.
Training data AUC AP
Real 0.992 0.990
GAN-sample1 0.951 0.923
GAN-sample2 0.956 0.935
GAN-sample1 + GAN-sample2 0.945 0.924
Real + Augment 0.991 0.988
Real + GAN-sample1 0.991 0.988
Real + GAN-sample2 0.989 0.983
Real + GAN-sample1 + GAN-sample2 0.989 0.983
Table 1: Performance of fracture classification models trained with different sets of training data. All Area under the receiver operating curve (AUC) and average precision (AP) scores are reported on a held-out test set of real fracture images that were absent from the training set for the classifiers and GAN.

4 Results

Results of our classification analysis is presented in Table 1. The highest test-set accuracy was achieved by training on real images (AUC 0.992), but all models achieved high performance, with models trained on GAN-sampled images alone achieving AUC .

Figure 1 depicts six randomly sampled (one time, with pre-specified seed) images from the GAN, along with top-1 near-neighbors in the feature space from the first two convolutional layers of a VGG network trained on fracture images. Top-1 neighbors from all other layers are in the appendix in Figure 6. No obvious copies are visible in any case. Figures 2-4 in the appendix depict t-SNE plots from 256 images sampled from each class from the real and first GAN-sampled training set. No obvious mode collapse is present, and a range of different types of fractures and annotation markers were learned by the GAN, though the samples appear less diverse than the true images.

5 Discussion

Visual inspection and classification results both indicate that our conditional GAN has learned key visual features that distinguish fracture from non-fracture images. Though of slightly lower quality than real images in appearance and performance, the objective success of classifiers trained solely on GAN samples is still striking and strongly implies that our GAN has learned clinical-relevant features sufficient to aid in machine learning. Based on these results, we are currently designing a simple trial to test the extent to which GAN-sampled images serve as an effective alternative to real images in human learning. Of note, while simulating the true data distribution is the goal of GAN training, in the setting of medical education, whether or not sampled images are indistinguishable from or as diverse as real images may or may not prove to fully correlated with whether they are helpful to students learning to internalize the key distinction between fractures from non-fractures.

We are particularly intrigued by the data augmentation results. Prior studies in the medical domain have reported improved classification performance using GANs as a means of data augmentation, however none of these in the setting of nearly perfect performance on real images alone (Santurkar et al., 2017; Li et al., 2018; Antoniou et al., 2017). These findings seem to suggest that GAN images may serve as effective augmentation in the low-data regime, but may be actively unhelpful in a high-data regime. We presently intend to conduct experiments simulating the relative impact of GAN augmentation when progressively limiting access to real images.

6 Acknowledgements

SGF was supported by training grants T32GM007753 and T15LM007092; the content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of General Medical Sciences or the National Institutes of Health.


  • (1)
  • Antoniou et al. (2017) Antreas Antoniou, Amos Storkey, and Harrison Edwards. 2017. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340 (2017).
  • Beers et al. (2018) Andrew Beers, James Brown, Ken Chang, J Peter Campbell, Susan Ostmo, Michael F Chiang, and Jayashree Kalpathy-Cramer. 2018. High-resolution medical image synthesis using progressively grown generative adversarial networks. arXiv preprint arXiv:1805.03144 (2018).
  • Chen and Koltun (2017) Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In IEEE International Conference on Computer Vision (ICCV), Vol. 1. 3.
  • Ching et al. (2018) Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al. 2018. Opportunities and obstacles for deep learning in biology and medicine. bioRxiv (2018), 142760.
  • Dar et al. (2018) Salman Ul Hassan Dar, Mahmut Yurt, Levent Karacan, Aykut Erdem, Erkut Erdem, and Tolga Çukur. 2018. Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks. arXiv preprint arXiv:1802.01221 (2018).
  • Frid-Adar et al. (2018) Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. 2018. Synthetic data augmentation using GAN for improved liver lesion classification. In Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on. IEEE, 289–293.
  • Gale et al. (2017) William Gale, Luke Oakden-Rayner, Gustavo Carneiro, Andrew P Bradley, and Lyle J Palmer. 2017. Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv preprint arXiv:1711.06504 (2017).
  • Gauthier (2014) Jon Gauthier. 2014. Conditional generative adversarial nets for convolutional face generation. Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition, Winter semester 2014, 5 (2014), 2.
  • Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.
  • Guibas et al. (2017) John T Guibas, Tejpal S Virdi, and Peter S Li. 2017. Synthetic Medical Images from Dual Generative Adversarial Networks. arXiv preprint arXiv:1709.01872 (2017).
  • Hinton and Roweis (2003) Geoffrey E Hinton and Sam T Roweis. 2003. Stochastic neighbor embedding. In Advances in neural information processing systems. 857–864.
  • Hu et al. (2017) Yipeng Hu, Eli Gibson, Li-Lin Lee, Weidi Xie, Dean C Barratt, Tom Vercauteren, and J Alison Noble. 2017. Freehand Ultrasound Image Simulation with Spatially-Conditioned Generative Adversarial Networks. In Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment. Springer, 105–115.
  • Karras et al. (2017) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).
  • Korkinof et al. (2018) Dimitrios Korkinof, Tobias Rijken, Michael O’Neill, Joseph Yearsley, Hugh Harvey, and Ben Glocker. 2018. High-resolution mammogram synthesis using progressive generative adversarial networks. arXiv preprint arXiv:1807.03401 (2018).
  • Li et al. (2018) Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. 2018. On the limitations of first order approximation in GAN dynamics. (2018).
  • Mirza and Osindero (2014) Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).
  • Nie et al. (2017) Dong Nie, Roger Trullo, Jun Lian, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. 2017. Medical image synthesis with context-aware generative adversarial networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 417–425.
  • Santurkar et al. (2017) Shibani Santurkar, Ludwig Schmidt, and Aleksander Madry. 2017. A Classification-Based Perspective on GAN Distributions. arXiv preprint arXiv:1711.00970 (2017).
  • Wang (2012) Shih-chang Wang. 2012. Competencies and Experiential Requirements in Radiology Training. In Radiology Education. Springer, 55–66.
  • Wolterink et al. (2017) Jelmer M Wolterink, Anna M Dinkla, Mark HF Savenije, Peter R Seevinck, Cornelis AT van den Berg, and Ivana Išgum. 2017. Deep MR to CT synthesis using unpaired data. In International Workshop on Simulation and Synthesis in Medical Imaging. Springer, 14–23.
  • Wu et al. (2018) Eric Wu, Kevin Wu, David Cox, and William Lotter. 2018. Conditional Infilling GANs for Data Augmentation in Mammogram Classification. In Image Analysis for Moving Organ, Breast, and Thoracic Images. Springer, 98–106.
  • Zhang et al. (2017) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. 2017. StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks. arXiv: 1710.10916 (2017).


Figure 2: t-SNE plot of additional examples of real, non-fractured images.
Figure 3: t-SNE plot of additional examples of GAN-generated, non-fractured images.
Figure 4: t-SNE plot of additional examples of real, fractured images.
Figure 5: t-SNE plot of additional examples of GAN-generated, fractured images.
Figure 6: t-SNE plot of additional examples of real and GANq images.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minumum 40 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description