Bilateral Filter: Graph Spectral Interpretation and Extensions
In this paper we study the bilateral filter proposed by Tomasi and Manduchi, as a spectral domain transform defined on a weighted graph. The nodes of this graph represent the pixels in the image and a graph signal defined on the nodes represents the intensity values. Edge weights in the graph correspond to the bilateral filter coefficients and hence are data adaptive. Spectrum of a graph is defined in terms of the eigenvalues and eigenvectors of the graph Laplacian matrix. We use this spectral interpretation to generalize the bilateral filter and propose more flexible and application specific spectral designs of bilateral-like filters. We show that these spectral filters can be implemented with -iterative bilateral filtering operations and do not require expensive diagonalization of the Laplacian matrix.
Bilateral filter, graph based signal processing, polynomial approximation
The bilateral filter (BF) proposed by Tomasi and Manduchi  has emerged as a powerful tool for adaptive processing of multidimensional data. Bilateral filtering smooths images while preserving edges, by taking the weighted average of the nearby pixels. The weights depend on both the spatial distance and photometric distance which provides local adaptivity to the given data. The bilateral filter and its variants are widely used in different applications such as denoising, edge preserving multi-scale decomposition, detail enhancement or reduction and segmentation etc. [2, 3, 4, 5, 6]. Bilateral filtering was developed as an intuitive tool without theoretical justification. Since then connections between the BF and other well known filtering frameworks such as anisotropic diffusion, weighted least squares, Bayesian methods, kernel regression and non-local means have been explored [7, 8, 9, 10, 11].
The BF is data dependent and hence a non-linear and non-shift invariant filter. So, it does not have a spectral interpretation in the traditional frequency domain of images. However we would like to have spectral interpretation so that we can modify the global properties of the signal by changing its frequency components. To overcome this difficulty, we view the BF as a vertex domain transform on a graph with pixels as vertices, intensity values of each node as graph signal and filter coefficients as link weights that capture the similarity between nodes. This graphical views of the BF is used in [12, 9, 13, 6, 14] but spectral design of filters for images using this graph has not been studied.
We can define spectral filters on these graphs where spectral response is calculated in terms of eigenvectors and eigenvalues of the graph Laplacian matrix [15, 16, 17, 18]. This spectral interpretation captures the oscillatory behavior of the graph signal  and thus allows us to extend of the concept of frequency to irregular domains. This has led to the design of frequency selective filtering operations on graphs similar to that in traditional signal processing. These graph spectral filters also have a vertex domain implementation.
In this paper, we interpret the BF as a -hop localized transform on the aforementioned graph. Because the link weights in the graph are data adaptive, the problem of structure preserving filtering boils down to low pass spectral filtering on that graph. We show that the BF can be characterized by a spectral response corresponding to a linear spectral decay. We also calculate the spectral response of iterated BF. We extend this novel insight to build better and more general bilateral-like filters using the machinery to design graph based transforms with desired spectral response. Our design allows one to choose the spectral response of the filter depending on the application which offers more flexibility. We also give a theoretical justification for the design using the framework of regularization on graphs. We show that these spectral filters do not require computationally expensive diagonalization of the graph Laplacian matrix and then provide an efficient algorithm for implementing these filters using the BF as a building block. We examine the performance of the proposed filters in a few applications.
Ii Bilateral Filter as a Graph Based Transform
Consider an input image to the BF. The value at each position in the output image is given by the weighted average of the pixels in .
The weights depend on both the euclidean and photometric distance between the pixels and . Let denote the position of the pixel . The weights are then given by
Spatial Gaussian weighting decreases the influence of distant pixels111Eq. (1) shows that the averaging is done over all pixels. However in practice, one assumes non-zero weights only for the pixels which have . and intensity Gaussian weighting decreases the influence of pixels with different intensities. Intuition is that only similar nearby pixels should get averaged so that blurring of edges is avoided.
Now, consider an undirected graph where the nodes are the pixels of the input image and the edges capture the similarity between two pixels as given by the BF weights (Figure 1). Image can be considered as a signal defined on this graph where the signal value at each node equals the corresponding pixel intensity.
It can be seen from (1) that the output at each node in the graph depends only on the nodes in its -hop neighborhood. So, the BF is a -hop localized graph based transform. Also, note that the BF includes the current pixel in the weighted average. So, the graph corresponding to the BF has a self loop i.e. an edge connecting each node to itself with weight . Other filtering techniques such as Gaussian smoothing and non-local means can also be described using similar graphical models [9, 12].
Ii-a Graph Spectrum and Data Adaptivity of the BF
Spectrum of a graph is defined in terms of the eigenvalues and eigenvectors of its Laplacian matrix. The combinatorial Laplacian matrix for the graph is defined . We use the normalized form of the Laplacian matrix given as . is a non-negative definite matrix . As a result has an orthogonal set of eigenvectors with corresponding eigenvalues . So, can be diagonalized as
Similar to classical Fourier transform, the eigenvectors and eigenvalues of the Laplacian matrix provide a spectral interpretation of the graph signals. The eigenvalues can be treated as graph frequencies, and are always situated in the interval on the real line. The eigenvectors of the Laplacian matrix demonstrate increasing oscillatory behavior as the magnitude of the graph frequency increases . The Graph Fourier Transform (GFT) of a signal is defined as its projection onto the eigenvectors of the graph, i.e., , or in matrix form . The inverse GFT is given by .
Figure 2 shows the fraction of total signal energy captured by the first spectral components of a graph signal corresponding to a image block. In this example we consider the following underlying graphs (1) the BF graph (2) the graph corresponding to Gaussian smoothing where the weights depend only on the geometric distance between two pixels. We can see that due to the data adaptivity of the BF graph, most of the signal energy is captured in low frequency part of the BF graph spectrum in comparison with spectrum of Gaussian smoothing graph. So, the spectral basis of the BF graph offers better energy compaction for the given signal.
Iii Spectral interpretation of the bilateral filter
Similar to conventional signal processing, graph spectral filtering is defined as
is the spectral response of the filter according to which spectral components of an input signal are modulated. Using the definition of GFT and the diagonalized form of , we can write graph spectral filtering in matrix notation as 
To exploit this framework of graph spectral filtering we rewrite the BF in (3) as
From this equation, we can see that the BF is a graph transform, similar to the one in (6), operating on the normalized input signal producing the normalized output . This normalization allows us to define the BF in terms of the non-negative definite matrix and thus have a spectral interpretation. It also ensures that a constant signal when normalized, is an eigenvector of associated with zero eigenvalue . Following (6) we have,
This shows that the BF is a frequency selective graph transform with a spectral response which corresponds to linear decay (See Figure 3).
The BF tries to preserve the low frequency components and attenuate the high frequency components.
The BF is used iteratively in many applications. There are two ways to iterate the BF (1) by changing the weights at each iteration using the result of previous iteration (2) by using fixed weights at each iteration as calculated from the initial image. In the first method the BF graph changes in every iteration. So we cannot have a spectral interpretation. In the second method the graph remains fixed at every iteration. Here we consider the second method. Iterating preserves strong edges while removing weaker details. This type of effect is desirable for applications such as stylization . The BF iterated -times can be written in matrix notation as
The spectral responses corresponding are shown in Figure 3. The figure suggests that iterative application of the BF suppresses more of the high frequency component which is consistent with the observation. Equations (8) and (10) give a different perspective to look at the bilateral filter. They hint at filter designs with better spectral responses that can be tailored to particular applications.
Iv Application specific spectral designs
The BF and iterated BF have fixed spectral responses. But these responses may not be suitable for all applications. Below we discuss two applications (1) image denoising and (2) segmentation to illustrate design of more flexible spectral filters.
We consider the problem of image denoising with additive zero-mean white noise.
where is the original image that we want to estimate, is the observed noisy image and is zero-mean white noise with variance . As explained before most of the signal energy lies in the low frequency part of the BF graph spectrum due to its data adaptivity. So most energy in the high frequency spectrum corresponds to noise. We can put this intuition in a more principled framework of regularization where the problem of denoising is equivalent to minimization of a penalty functional . This penalty functional is composed of two terms. The first term is a fit measure and the second term is a data dependent smoothness constraint as captured by regularization operators on the BF graph [22, 23].
Note that we normalize and as in (7). The regularization functional can be written in spectral domain as
is chosen to be a non-decreasing function in so that the high frequency components are penalized more strongly. Putting we get the optimal as
So, for a chosen regularization functional , the spectral response of the denoising filter is given by
Denoising filters suggested by the regularization framework are essentially low pass filters in the spectral domain of the graph (Figure 5). The BF has a relatively poor spectral decay profile due to which it blurs the textures in the denoising process.
Shi and Malik  formulated the problem of image segmentation as a graph partitioning problem on a graph similar to the BF graph. To obtain an -way partition of the graph, we only need to find the projection of the graph signal on the first few eigenvectors with the smallest eigenvalues. This projection gives a very coarse version of the signal which then can be used to perform image segmentation. So a suitable filter for this application should be a low pass filter in the graph spectral domain with a small cut off frequency and sharp transition band (Figure 6). Iterative application of the BF also gives a coarse version of the image with details removed. But it favours the eigenvector with the smallest eigenvalue  which corresponds to the DC component of the image as seen from its spectral response. On the other hand, for graph partitioning we need the eigenvector with second smallest eigenvalue (and eigenvectors with larger eigenvalues for finer partitions).
To summarize, different applications require filters having different spectral responses . So, we would like to design more general bilateral-like filters with desired spectral response. These filters are of the form . A direct implementation of these filters requires diagonalization of which is of the order . For large graphs such as the one considered here, this is computationally very expensive. Fortunately, we can approximate any spectral response by a polynomial in . These polynomial spectral filters are -hop localized on the graph where is the degree of the polynomial. This leads to an easy and efficient implementation scheme for these filters as explained in the next section.
V Polynomial approximation and fast implementation
where the roots can be either real or complex conjugate pairs. This generalization allows degrees of freedom in choosing the spectral response of the filter. Further, the corresponding transform in pixel domain is a matrix polynomial of with the same roots as in (16) i.e.,
This leads to following result:
Any graph filter on an image having a polynomial spectral response of degree can be implemented in the pixel domain as an iterative step bilateral filter operation.
Further, if is not a polynomial of , it can be approximated with a polynomial kernel which can then be implemented as a generalized iterative bilateral filtering operation. It has been shown in  that minimax polynomial approximation of any kernel not only minimizes the Chebychev norm (worst-case norm) of the error between kernel and its approximation, it also minimizes the upper-bound on the error between exact and approximated filters. In our experiments, we approximate any non-polynomial with the truncated Chebychev polynomials (which are a good approximation of minimax polynomials).
We examine the performance of the proposed spectral design of bilateral-like filters in two applications. First, we consider the image denoising problem. We experiment with a spectral response obtained by the regularization framework in Section 4. We take the regularization functional which suggests a denoising filter with . We take its degree polynomial approximation. Figure 5 shows the denoising results using this filter and the BF 222The BF is not the best denoising filter available. We use it in our comparison to emphasize the qualitative differences in filtering results.. It can be seen that the BF preserves edges, but it blurs the texture in the denoising process while the proposed denoising filter does a better job of preserving texture. This is also reflected in the SNR values.
Next, we consider an iterative application of the BF. Iterated BF removes minor details from the image while preserving prominent edges. This can be used as an effective preprocessing step in edge-detection and segmentation etc. As stated before, iterated BF favours the DC component which is not useful for segmentation. We use a low pass spectral kernel with small cut-off and sharp transition band so that the second (and a few higher) spectral components get at least as much weight as the DC component. We use a degree polynomial approximation of this kernel and compare it with iterations of the BF. Figure 6 shows that weak edges are blurred more with iterated BF (as expected from the spectral response) compared to the proposed filter.
In this paper we explained the bilateral filter as a graph spectral filtering operation. With this novel perspective, we proposed a family of more flexible bilateral-like filters with desired spectral responses. We gave an easy implementation scheme for these filters. Their utility was motivated through few examples. An immediate interesting extension to this work would be to explore different spectral filters suitable for particular applications. Another topic of interest is the design of filter banks using these bilateral-like filters.
-  C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision, Jan 1998, pp. 839–846.
-  S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, “Bilateral filtering: Theory and applications,” Foundations and Trends in Computer Graphics and Vision, vol. 4, no. 1, 2009.
-  Ming Zhang and B. K. Gunturk, “Multiresolution bilateral filtering for image denoising,” Image Processing, IEEE Transactions on, vol. 17, no. 12, pp. 2324 –2333, Dec 2008.
-  R. Fattal, M. Agrawala, and S. Rusinkiewicz, “Multiscale shape and detail enhancement from multi-light image collections,” ACM Trans. Graph., vol. 26, no. 3, 2007.
-  F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” ACM Trans. Graph., vol. 21, no. 3, pp. 257–266, July 2002.
-  C. Ye, Y. Lin, M. Song, C. Chen, and D. Jacobs, “Spectral graph cut from a filtering point of view,” arXiv:1205.4450v2 [cs.CV], 2012.
-  M. Elad, “On the origin of the bilateral filter and ways to improve it,” IEEE Transactions on Image Processing, vol. 11, no. 10, pp. 1141–1151, Oct 2002.
-  D. Barash, “Fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 844–847, 2002.
-  P. Milanfar, “A tour of modern image filtering: New insights and methods, both practical and theoretical,” IEEE Signal Processing Magazine, vol. 30, no. 1, pp. 106–128, Jan. 2013.
-  A. Singer, Y. Shkolnisky, and B. Nadler, “Diffusion interpretation of nonlocal neighborhood filters for signal denoising,” SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 118–139, 2009.
-  A. Buades, B. Coll, and J.M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Modeling & Simulation, vol. 4, no. 2, pp. 490–530, 2005.
-  G. Peyré, “Image processing with nonlocal spectral bases,” SIAM Journal on Multiscale Modeling and Simulation, vol. 7, no. 2, pp. 703–730, 2008.
-  G. Noel, K. Djouani, B. Van Wyk, and Y. Hamam, “Bilateral mesh filtering,” Pattern Recognition Letters, vol. 33, no. 9, pp. 1101–1107, 2012.
-  C. Jian, K. Inoue, K. Hara, and K. Urahama, “Fixed-coefficient iterative bilateral filters for graph-based image processing,” in Advances in Image and Video Technology, vol. 5414 of Lecture Notes in Computer Science, pp. 473–484. Springer, 2009.
-  D. Hammond, P. Vandergheynst, and R. Gribonval, “Wavelets on graphs via spectral graph theory,” Applied and Computational Harmonic Analysis, vol. 30, no. 2, pp. 129 – 150, 2011.
-  S.K. Narang and A. Ortega, “Perfect reconstruction two-channel wavelet filter banks for graph structured data,” IEEE Transactions on Signal Processing, vol. 60, no. 6, pp. 2786–2799, June 2012.
-  D. Shuman, S. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, “Signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular data domains,” to appear in IEEE Signal Processing Magazine, 2013.
-  A. Sandryhaila and J. M. F. Moura, “Discrete signal processing on graphs,” arXiv:1210.4752v2 [cs.SI], 2012.
-  E.B. Davies, G.M.L. Gladwell, J. Leydold, and P.F. Stadler, “Discrete nodal domain theorems,” Linear Algebra and its Applications, vol. 336, no. 1, pp. 51–60, 2001.
-  F.R.K. Chung, Spectral graph theory, vol. 92, CBMS Regional Conference Series in Mathematics, AMS, 1997.
-  S.K. Narang and A. Ortega, “Compact support biorthogonal wavelet filterbanks for arbitrary undirected graphs,” arXiv:1210.8129 [cs.IT], 2012.
-  A. Smola and R. Kondor, “Kernels and regularization on graphs,” in Proceedings of the Annual Conference on Computational Learning Theory. 2003, Lecture Notes in Computer Science, Springer.
-  D. Zhou and B. Schölkopf, “A regularization framework for learning from graph data,” in ICML Workshop on Statistical Relational Learning, 2004, pp. 132–137.
-  J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888–905, Aug 2000.