# Probabilistic Intra-Retinal Layer Segmentation in 3-D OCT Images Using Global Shape Regularization

###### Abstract

With the introduction of spectral-domain optical coherence tomography (OCT), resulting in a significant increase in acquisition speed, the fast and accurate segmentation of 3-D OCT scans has become evermore important. This paper presents a novel probabilistic approach, that models the appearance of retinal layers as well as the global shape variations of layer boundaries. Given an OCT scan, the full posterior distribution over segmentations is approximately inferred using a variational method enabling efficient probabilistic inference in terms of computationally tractable model components: Segmenting a full 3-D volume takes around a minute. Accurate segmentations demonstrate the benefit of using global shape regularization: We segmented 35 fovea-centered 3-D volumes with an average unsigned error of as well as 80 normal and 66 glaucomatous 2-D circular scans with errors of and respectively. Furthermore, we utilized the inferred posterior distribution to rate the quality of the segmentation, point out potentially erroneous regions and discriminate normal from pathological scans. No pre- or postprocessing was required and we used the same set of parameters for all data sets, underlining the robustness and out-of-the-box nature of our approach.

###### keywords:

Statistical shape model, Retinal layer segmentation, Pathology detection, Optical coherence tomography^{†}

^{†}journal: Medical Image Analysis

## 1 Introduction

Optical coherence tomography (OCT) is an in vivo imaging technique, measuring the delay and magnitude of backscattered light. Providing micrometer resolution and millimeter penetration depth into retinal tissue (drexler2008), OCT is well suited for ophthalmic imaging. Since no other method can perform noninvasive imaging with such a resolution, OCT has become a standard in clinical ophthalmology (schuman2004). Several studies showed the applicability for the diagnosis of pathologies such as glaucoma or age-related macular degeneration (bowd2001; zysk2007). The recent introduction (deboer2003; wojtkowski2002) of spectral-domain OCT dramatically increased the imaging speed and enabled the acquisition of 3-D volumes containing hundreds of B-scans. Since manual segmentation of retinal layers is tedious and time-consuming, automated segmentation becomes evermore important given the growing amount of gathered data. Furthermore, a probabilistic model that enables to infer uncertainties of estimates, provides essential information for practitioners, in addition to the segmentation result.

Various approaches for the task of retina segmentation in OCT images were published. All have in common that they generate appearance terms based either on intensity or gradient information. On top of that regularization is applied, which makes predictions more robust to speckle noise or shadowing caused by blood vessels. In order to provide a systematic overview over this vast field of approaches, we choose to distinguish them by the method used for regularization.

One major class is composed of rule-based heuristic techniques (ahlers2008; fernandez2005; ishikawa2005; mayer2010), which for example apply outlier detection along with linear interpolation to account for erroneous segmentations. Other approaches (baroni2007; yang2010) use dynamic programming for single Markov chains per boundary and constrain the maximal vertical distance between neighboring boundary positions. vermeer2011 classify pixels using support vector machines and regularize the output using level-set techniques. None of these approaches incorporates shape prior information.

Active contour approaches include gradient respectively intensity-based methods (mishra2009; yazdanpanah2009; yazdanpanah2011). yazdanpanah2009; yazdanpanah2011 augment the classical active contour functional by a simple circular shape prior. All three approaches were only tested on OCT-scans that exclude the foveal region, thus contain mainly flat boundaries with rather simple shapes.

A series of more advanced approaches (antony2010; dufour2013; garvin2009; song2013) construct a geometric graph to simultaneously segment all boundaries in a 3-D OCT volume. Unlike previously presented approaches, they take into account the interaction of neighboring boundaries to mutually restrict their relative positions. This shape prior information is encoded into the graph as hard constraints (antony2010; garvin2009) or, as recently introduced by song2013 and subsequently extended by dufour2013, as probabilistic soft constraints. However, due to computational limitations, only local shape information is included and boundaries are segmented in stages.

Finally, kajic2010 apply the popular active appearance models that match statistical models for appearance and shape, to a given OCT scan.
Although non-local shape modeling is in the scope of their approach, they only use landmarks, i.e. sparsely sampled boundary positions instead of the full shape model.
Furthermore, only a maximum likelihood point estimate is inferred, instead of a distribution over shapes.

Contribution. We present a novel probabilistic approach for the OCT retina segmentation problem. Our probabilistic graphical model combines appearance models with a global shape prior, that comprises local as well as long-range interactions between boundaries. The discrete part of the model features a highly parallelizable column-wise discrete segmentation, that nevertheless takes into account all other image columns. In order to infer the posterior probability of this model, we utilize variational inference, a deterministic approximation framework.

To our knowledge this is the only work, where a full global shape prior is employed for the task of OCT retina segmentation. Moreover, we are not aware of any other segmentation approach that infers a full probability distribution. Our approach offers excellent segmentation performance, outperforming approaches relying on local or no shape regularization, as well as pathology detection and an assessment of segmentation quality. Fig. 1 illustrates the segmented boundaries, but additional boundaries like the external limiting membrane (ELM) could easily be incorporated if ground truth is available.

This work evolved out of preliminary ideas presented in a previous conference paper (rathke2011b).

Organization. The next section will introduce our probabilistic graphical model. Section LABEL:chap:inference
evaluates the posterior distribution via variational inference, and we solve the corresponding optimization problem in Section LABEL:chap:optimization in terms of efficiently solvable convex subproblems. Section LABEL:chap:exp-methods and LABEL:chap:results present the data sets we used for evaluation and the corresponding results. We conclude in Section LABEL:chap:discussion with a discussion and possible directions for future work.

## 2 Graphical Model

This section presents our probabilistic graphical model, statistically modeling an OCT scan and its segmentations and respectively. We introduce , the discretized version of the continuous boundary vector , to make mathematically explicit the connection between the discrete pixel domain of and the continuous boundary domain of . Our ansatz is given by

(1) |

where the factors are