Unsupervised Anomaly Detection for X-Ray Images

Deep Learning techniques are ubiquitous and achieving state-of-the-art performance in many areas. However, they require vast amounts of labeled data as witnessed by the marvelous boost in image recognition after the publication of the large scale ImageNet data set [4, 11]. In medical applications, labels are expensive to acquire. While anyone can decide whether an image depicts a dog or a cat, deciding whether a medical image shows abnormalities, is a highly difficult task requiring specialists with years of training. Another specialty of medical applications is that a simple classification decision does often not suffice. End-to-end deep learning solutions tend to be hard to interpret, preventing their application in an area as sensitive as deciding for a treatment. Moreover, additional patient information such as the patient’s medical history, and clinical test results are often crucial to a correct diagnosis. Integrating this information into an end-to-end pipeline is difficult and makes results even less interpretable. Thus, the motivation for our work is to let doctors decide about the final diagnosis and treatment and develop a system, which can provide hints for doctors where to pay more attention to.

Hence, in this work, we investigate how we can support doctors to faster assess X-ray images, and reduce the chance of overlooking suspicious regions. To this end, we demonstrate how state-of-the-art unsupervised methods, such as Autoencoders (AE) or Generative Adversarial Networks (GANs), can be used for anomaly detection on X-ray images. As this dataset is noisy, and this is a general problem for a lot of real-world datasets, we present a sophisticated preprocessing pipeline to obtain better training data. Afterwards, we train several unsupervised models, and explain for each, how to obtain several image-level anomaly scores. For some of them, it is even natural to obtain pixel-wise annotations, highlighting anomalous regions. One of our main findings is that accurate data preprocessing is indispensable. The advantage of using autoencoders is that they naturally can provide pixel-level anomaly heatmaps, which can be used to understand model decisions. In contrast, GAN-based approaches seem to be able to cope with more noisy data, yet being only able to produce image-wise anomaly scores. We envision that this methodology can be easily installed in clinical daily routine to support doctors in quickly assessing X-ray images and spotting candidate regions for anomalies.

In this work, we focus on a subset of the MURA dataset [18] containing only hand images. In total, we have 5,543 images of 2,018 studies of 1,945 patients. Each study is labeled as negative or positive, where positive means that there was an anomaly diagnosed in this study. There are 521 positive studies, with a total of 1,484 images. Figure 1 shows some examples from the dataset. In summary, our contributions are as follows:

1. We present a powerful preprocessing pipeline for the MURA dataset [18], enabling the construction of a high-quality training set.

2. We extensively survey unsupervised Deep Learning methods, and present approaches on how to obtain image-level and even pixel-level anomaly scores.

3. We show extensive experiments on a real-world dataset evaluating the influence of proper preprocessing as well as the usability of the anomaly scores. To foster reproducibility, we will make our code public in the camera-ready version.

The rest of the paper is structured as follows: In Section 2 we describe our approach. We start with the description of data preprocessing in 2.1 and describe anomaly detection approaches along with anomaly scores in section 2.2. We discuss related work in section 3. Finally, Section 4 shows quantitative and qualitative experimental results on image-level and pixel-level anomaly detection.

With the rapid advancement of deep learning methods, they have also found their way into medical imaging, cf. e.g. [12, 19]. Despite the limited availability of labels in medical contexts, supervised methods make up the vast majority. Very likely, this is due to the easier trainability, but possibly also because the interpretability of the results so far has often been secondary. Sato et al. [22] use a 3D CAE for a pathology detection method in CT scans of brains. The CAE is trained solely on normal images, and at test time, the MSE between the image and its reconstruction is taken as the anomaly score. Uzunova et al. [25] use VAE for medical 2D and 3D CT images. Similarly, they use MSE reconstruction loss as the anomaly score. Besides the KL-divergence in latent space, they use a $L_1$ reconstruction loss for training, which produced less smooth output. GANomaly [2] and its extension with skip-connections uses an AE and maps the reconstructed input back to the latent space. The anomaly score is computed in latent space between original and reconstructed input. They apply their methods on X-Ray security imagery to detect anomalous items in baggage. Recently, there have been a lot of publications using the currently popular GANs. For example, [14] uses a semi-supervised approach for anomaly detection in chest X-ray images. They replace the standard discriminator classification into real and fake, with a three-way classification into real-normal, real-abnormal, and fake. While this allows training with fewer labels, it still requires them for training. Schlegl et al [23], train a DC-GAN on slices of OCT scans, where the original volume is cut along the x-z axis, and the slices are further randomly cropped. At test time, they use gradient descent to iteratively solve the inverse problem of obtaining a latent vector that produces the image. Stopping after a few iterations, the $L_1$ distance between the generated image and the input image is considered as residual loss. To summarize, the focus of recent work for anomaly detection approaches lies either in applying existing methods for a new type of data or adapting unsupervised methods for anomaly detection. Instead, we provide an extensive evaluation of state-of-the-art unsupervised learning approaches that can be directly used for anomaly detection. Furthermore, we evaluate the importance of different preprocessing steps and compare methods with regard to explainability.

We demonstrate the capability of our preprocessing pipeline and all described models in experiments on a subset of the MURA dataset containing only X-ray images of hands. 3,062 images are stored in a single-channel PNG image, and 2,481 are stored with three RGB channels. However, all images look like gray-scale images, which is why we convert all 3-channel images to a single channel. The longest side of the images is always 512 pixels in size. The smaller side ranges from 160 to 512, with the majority between 350 and 450.

As our approach is unsupervised, we train only on negative images, i.e. images without an anomaly. Furthermore, to avoid test leakage, we split the data by patient, and not by study or image, to ensure that we do not have an image of a patient in the training data, and another image of the same patient in the test or validation data. To this end, we proceed as follows: Let $P$ be the set of all patients, and $P^{+}$ be the set of patients with a study that is labeled as abnormal. The rest of the patients is denoted by $P^{-}:=P\setminus P^{+}$. For the test and validation set, we aim at having balanced classes. Therefore, we distribute $P^{+}$ evenly at random across test and validation. Afterwards, we randomly sample the same number of patients without known anomalies for test and validation and use the rest of the patients for training. The procedure is visualized in Figure 4. In total, we end up with 2,554 training images, 1,494 validation images, and 1,495 test images.

We trained all models on a machine with one NVIDIA Tesla V100 GPU with 16GiB of VRAM, 20 cores and 360GB of RAM. Following [17], we train our models from scratch and do not use transfer learning from large image classification datasets. We performed a manual hyper-parameter search on the validation set and selected the best-performing models per type with respect to Area-under-Curve for the Receiver-Operator-Curve (ROC-AUC). We report the ROC-AUC on the test set.

In this paper, we investigated methods for unsupervised anomaly detection in X-ray images. To this end, we surveyed two families of unsupervised models, auto-encoders and GANs, regarding their applicability to derive anomaly scores. In addition, we provide a sophisticated multi-step preprocessing pipeline. In our experiments, we compare the methods against each other, and furthermore, reveal that the preprocessing is crucial for most models to obtain good results on real-world data. For the auto-encoder family, we study the interpretability of pixel-wise losses as anomaly heatmap and verify that in cases of anomalies which a non-expert can detect (e.g. metal pieces in the hand), these heatmaps closely match the anomalous regions. As future work, we envision the extension to broader datasets such as the full MURA dataset, as well as obtaining pixel-level anomaly scores for the GAN based models. To this end, methods from the field of explainable AI, such as grad-CAM [24] or LRP [3] can be applied to the discriminator to obtain heatmaps similarly to those of the AE models. Moreover, we see the potential for different model architectures closer tailored to the specific problem and data type, as well as the possibility of building an ensemble model using the different ways how to extract anomaly scores from single models, or even across different model types.

We would like to thank Franz Pfister and Rami Eisaway from deepc (www.deepc.ai) for access to the data and support in understanding the use case. Part of this work has been conducted during a practical course at Ludwig-Maximilians-Unversität München funded by Z.DB. This work has been funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibilities for its content.