Convolutional neural network architecture for geometric matching

Convolutional neural network architecture for geometric matching

Abstract

We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine or thin-plate spline transformation, and estimating its parameters. The contributions of this work are three-fold. First, we propose a convolutional neural network architecture for geometric matching. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from /media/arxiv_projects/37807/synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never seen before images. Finally, we show that the same model can perform both instance-level and category-level matching giving state-of-the-art results on the challenging Proposal Flow dataset.

1Introduction

Estimating correspondences between images is one of the fundamental problems in computer vision [20] with applications ranging from large-scale 3D reconstruction [2] to image manipulation [22] and semantic segmentation [44]. Traditionally, correspondences consistent with a geometric model such as epipolar geometry or planar affine transformation, are computed by detecting and matching local features (such as SIFT [40] or HOG [12]), followed by pruning incorrect matches using local geometric constraints [45] and robust estimation of a global geometric transformation using algorithms such as RANSAC [19] or Hough transform [34]. This approach works well in many cases but fails in situations that exhibit (i) large changes of depicted appearance due to intra-class variation [23], or (ii) large changes of scene layout or non-rigid deformations that require complex geometric models with many parameters which are hard to estimate in a manner robust to outliers.

In this work we build on the traditional approach and develop a convolutional neural network (CNN) architecture that mimics the standard matching process. First, we replace the standard local features with powerful trainable convolutional neural network features [33], which allows us to handle large changes of appearance between the matched images. Second, we develop trainable matching and transformation estimation layers that can cope with noisy and incorrect matches in a robust way, mimicking the good practices in feature matching such as the second nearest neighbor test [40], neighborhood consensus [45] and Hough transform-like estimation [34].

The outcome is a convolutional neural network architecture trainable for the end task of geometric matching, which can handle large appearance changes, and is therefore suitable for both instance-level and category-level matching problems.

Figure 1: Our trained geometry estimation network automatically aligns two images with substantial appearance differences. It is able to estimate large deformable transformations robustly in the presence of clutter.
Figure 1: Our trained geometry estimation network automatically aligns two images with substantial appearance differences. It is able to estimate large deformable transformations robustly in the presence of clutter.

2Related work

The classical approach for finding correspondences involves identifying interest points and computing local descriptors around these points [25]. While this approach performs relatively well for instance-level matching, the feature detectors and descriptors lack the generalization ability for category-level matching.

Recently, convolutional neural networks have been used to learn powerful feature descriptors which are more robust to appearance changes than the classical descriptors [29]. However, these works still divide the image into a set of local patches and extract a descriptor individually from each patch. Extracted descriptors are then compared with an appropriate distance measure [29], by directly outputting a similarity score [24], or even by directly outputting a binary matching/non-matching decision [3].

In this work, we take a different approach, treating the image as a whole, instead of a set of patches. Our approach has the advantage of capturing the interaction of the different parts of the image in a greater extent, which is not possible when the image is divided into a set of local regions.

Related are also network architectures for estimating inter-frame motion in video [52] or instance-level homography estimation [14], however their goal is very different from ours, targeting high-precision correspondence with very limited appearance variation and background clutter. Closer to us is the network architecture of [30] which, however, tackles a different problem of fine-grained category-level matching (different species of birds) with limited background clutter and small translations and scale changes, as their objects are largely centered in the image. In addition, their architecture is based on a different matching layer, which we show not to perform as well as the matching layer used in our work.

Some works, such as [10], have addressed the hard problem of category-level matching, but rely on traditional non-trainable optimization for matching [10], or guide the matching using object proposals [23]. On the contrary, our approach is fully trainable in an end-to-end manner and does not require any optimization procedure at evaluation time, or guidance by object proposals.

Others [35] have addressed the problems of instance and category-level correspondence by performing joint image alignment. However, these methods differ from ours as they: (i) require class labels; (ii) don’t use CNN features; (iii) jointly align a large set of images, while we align image pairs; and (iv) don’t use a trainable CNN architecture for alignment as we do.

3Architecture for geometric matching

In this section, we introduce a new convolutional neural network architecture for estimating parameters of a geometric transformation between two input images. The architecture is designed to mimic the classical computer vision pipeline ( [42]), while using differentiable modules so that it is trainable end-to-end for the geometry estimation task. The classical approach consists of the following stages: (i) local descriptors ( SIFT) are extracted from both input images, (ii) the descriptors are matched across images to form a set of tentative correspondences, which are then used to (iii) robustly estimate the parameters of the geometric model using RANSAC or Hough voting.

Figure 2: Diagram of the proposed architecture. Images I_A and I_B are passed through feature extraction networks which have tied parameters W, followed by a matching network which matches the descriptors. The output of the matching network is passed through a regression network which outputs the parameters of the geometric transformation.
Figure 2: Diagram of the proposed architecture. Images and are passed through feature extraction networks which have tied parameters , followed by a matching network which matches the descriptors. The output of the matching network is passed through a regression network which outputs the parameters of the geometric transformation.

Our architecture, illustrated in Figure 2, mimics this process by: (i) passing input images and through a siamese architecture consisting of convolutional layers, thus extracting feature maps and which are analogous to dense local descriptors, (ii) matching the feature maps (“descriptors”) across images into a tentative correspondence map , followed by a (iii) regression network which directly outputs the parameters of the geometric model, , in a robust manner. The inputs to the network are the two images, and the outputs are the parameters of the chosen geometric model, a 6-D vector for an affine transformation.

In the following, we describe each of the three stages in detail.

3.1Feature extraction

The first stage of the pipeline is feature extraction, for which we use a standard CNN architecture. A CNN without fully connected layers takes an input image and produces a feature map , which can be interpreted as a dense spatial grid of -dimensional local descriptors. A similar interpretation has been used previously in instance retrieval [6] demonstrating high discriminative power of CNN-based descriptors. Thus, for feature extraction we use the VGG-16 network [48], cropped at the pool4 layer (before the ReLU unit), followed by per-feature L2-normalization. We use a pre-trained model, originally trained on ImageNet [13] for the task of image classification. As shown in Figure 2, the feature extraction network is duplicated and arranged in a siamese configuration such that the two input images are passed through two identical networks which share parameters.

3.2Matching network

The image features produced by the feature extraction networks should be combined into a single tensor as input to the regressor network to estimate the geometric transformation. We first describe the classical approach for generating tentative correspondences, and then present our matching layer which mimics this process.

Classical methods start by computing similarities between all pairs of descriptors across the two images. From this point on, the original descriptors are discarded as all the necessary information for geometry estimation is contained in the pairwise descriptor similarities and their spatial locations. Secondly, the pairs are pruned by either thresholding the similarity values, or, more commonly, only keeping the matches which involve the nearest (most similar) neighbors. Furthermore, the second nearest neighbor test [40] prunes the matches further by requiring that the match strength is significantly stronger than the second best match involving the same descriptor, which is very effective at discarding ambiguous matches.

Our matching layer applies a similar procedure. Analogously to the classical approach, only descriptor similarities and their spatial locations should be considered for geometry estimation, and not the original descriptors themselves.

To achieve this, we propose to use a correlation layer followed by normalization. Firstly, all pairs of similarities between descriptors are computed in the correlation layer. Secondly, similarity scores are processed and normalized such that ambiguous matches are strongly down-weighted.

Figure 3: Correlation map computation with CNN features. The correlation map c_{AB} contains all pairwise similarities between individual features \mathbf{f}_A\in f_A and \mathbf{f}_B\in f_B. At a particular spatial location (i,j) the correlation map output c_{AB} contains all the similarities between \mathbf{f}_B(i,j) and all \mathbf{f}_A\in f_A.
Figure 3: Correlation map computation with CNN features. The correlation map contains all pairwise similarities between individual features and . At a particular spatial location the correlation map output contains all the similarities between and all .

In more detail, given L2-normalized dense feature maps , the correlation map outputted by the correlation layer contains at each position the scalar product of a pair of individual descriptors and , as detailed in Eq. .

where and indicate the individual feature positions in the dense feature maps, and is an auxiliary indexing variable for .

A diagram of the correlation layer is presented in Figure 3. Note that at a particular position , the correlation map contains the similarities between at that position and all the features of .

As is done in the classical methods for tentative correspondence estimation, it is important to postprocess the pairwise similarity scores to remove ambiguous matches. To this end, we apply a channel-wise normalization of the correlation map at each spatial location to produce the final tentative correspondence map . The normalization is performed by ReLU, to zero out negative correlations, followed by L2-normalization, which has two desirable effects. First, let us consider the case when descriptor correlates well with only a single feature in . In this case, the normalization will amplify the score of the match, akin to the nearest neighbor matching in classical geometry estimation. Second, in the case of the descriptor matching multiple features in due to the existence of clutter or repetitive patterns, matching scores will be down-weighted similarly to the second nearest neighbor test [40]. However, note that both the correlation and the normalization operations are differentiable with respect to the input descriptors, which facilitates backpropagation thus enabling end-to-end learning.

The first step of our matching layer, namely the correlation layer, is somewhat similar to layers used in DeepMatching [52] and FlowNet [18]. However, DeepMatching [52] only uses deep RGB patches and no part of their architecture is trainable. FlowNet [18] uses a spatially constrained correlation layer such that similarities are are only computed in a restricted spatial neighborhood thus limiting the range of geometric transformations that can be captured. This is acceptable for their task of learning to estimate optical flow, but is inappropriate for larger transformations that we consider in this work. Furthermore, neither of these methods performs score normalization, which we find to be crucial in dealing with cluttered scenes.

Previous works have used other matching layers to combine descriptors across images, namely simple concatenation of descriptors along the channel dimension [14] or subtraction [30]. However, these approaches suffer from two problems. First, as following layers are typically convolutional, these methods also struggle to handle large transformations as they are unable to detect long-range matches. Second, when concatenating or subtracting descriptors, instead of computing pairwise descriptor similarities as is commonly done in classical geometry estimation and mimicked by the correlation layer, image content information is directly outputted. To further illustrate why this can be problematic, consider two pairs of images that are related with the same geometric transformation – the concatenation and subtraction strategies will produce different outputs for the two cases, making it hard for the regressor to deduce the geometric transformation. In contrast, the correlation layer output is likely to produce similar correlation maps for the two cases, regardless of the image content, thus simplifying the problem for the regressor. In line with this intuition, in Sec.  ? we show that the concatenation and subtraction methods indeed have difficulties generalizing beyond the training set, while our correlation layer achieves generalization yielding superior results.

3.3Regression network

The normalized correlation map is passed through a regression network which directly estimates parameters of the geometric transformation relating the two input images. In classical geometry estimation, this step consists of robustly estimating the transformation from the list of tentative correspondences. Local geometric constraints are often used to further prune the list of tentative matches [45] by only retaining matches which are consistent with other matches in their spatial neighborhood. Final geometry estimation is done by RANSAC [19] or Hough voting [34].

We again mimic the classical approach using a neural network, where we stack two blocks of convolutional layers, followed by batch normalization [27] and the ReLU non-linearity, and add a final fully connected layer which regresses to the parameters of the transformation, as shown in Figure 4. The intuition behind this architecture is that the estimation is performed in a bottom-up manner somewhat like Hough voting, where early convolutional layers vote for candidate transformations, and these are then processed by the later layers to aggregate the votes. The first convolutional layers can also enforce local neighborhood consensus [45] by learning filters which only fire if nearby descriptors in image A are matched to nearby descriptors in image B, and we show qualitative evidence in Sec.  ? that this indeed does happen.

Figure 4: Architecture of the regression network. It is composed of two convolutional layers without padding and stride equal to 1, followed by batch normalization and ReLU, and a final fully connected layer which regresses to the P transformation parameters.
Figure 4: Architecture of the regression network. It is composed of two convolutional layers without padding and stride equal to 1, followed by batch normalization and ReLU, and a final fully connected layer which regresses to the transformation parameters.

A potential alternative to a convolutional regression network is to use fully connected layers. However, as the input correlation map size is quadratic in the number of image features, such a network would be hard to train due to a large number of parameters that would need to be learned, and it would not be scalable due to occupying too much memory and being too slow to use. It should be noted that even though the layers in our architecture are convolutional, the regressor can learn to estimate large transformations. This is because one spatial location in the correlation map contains similarity scores between the corresponding feature in image B and all the features in image A (c.f. equation ), and not just the local neighborhood as in [18].

3.4Hierarchy of transformations

Another commonly used approach when estimating image to image transformations is to start by estimating a simple transformation and then progressively increase the model complexity, refining the estimates along the way [39]. The motivation behind this method is that estimating a very complex transformation could be hard and computationally inefficient in the presence of clutter, so a robust and fast rough estimate of a simpler transformation can be used as a starting point, also regularizing the subsequent estimation of the more complex transformation.

We follow the same good practice and start by estimating an affine transformation, which is a 6 degree of freedom linear transformation capable of modeling translation, rotation, non-isotropic scaling and shear. The estimated affine transformation is then used to align image B to image A using an image resampling layer [28]. The aligned images are then passed through a second geometry estimation network which estimates 18 parameters of a thin-plate spline transformation. The final estimate of the geometric transformation is then obtained by composing the two transformations, which is also a thin-plate spline. The process is illustrated in Figure 5.

Figure 5: Estimating progressively more complex geometric transformations. Images A and B are passed through a network which estimates an affine transformation with parameters \hat{\theta}_{\text{Aff}} (see Fig. ). Image A is then warped using this transformation to roughly align with B, and passed along with B through a second network which estimates a thin-plate spline (TPS) transformation that refines the alignment.
Figure 5: Estimating progressively more complex geometric transformations. Images A and B are passed through a network which estimates an affine transformation with parameters (see Fig. ). Image A is then warped using this transformation to roughly align with B, and passed along with B through a second network which estimates a thin-plate spline (TPS) transformation that refines the alignment.

4Training

In order to train the parameters of our geometric matching CNN, it is necessary to design the appropriate loss function, and to use suitable training data. We address these two important points next.

4.1Loss function

We assume a fully supervised setting, where the training data consists of pairs of images and the desired outputs in the form of the parameters of the ground-truth geometric transformation. The loss function is designed to compare the estimated transformation with the ground-truth transformation and, more importantly, compute the gradient of the loss function with respect to the estimates . This gradient is then used in a standard manner to learn the network parameters which minimize the loss function by using backpropagation and Stochastic Gradient Descent.

It is desired for the loss to be general and not specific to a particular type of geometric model, so that it can be used for estimating affine, homography, thin-plate spline or any other geometric transformation. Furthermore, the loss should be independent of the parametrization of the transformation and thus should not directly operate on the parameter values themselves. We address all these design constraints by measuring loss on an imaginary grid of points which is being deformed by the transformation. Namely, we construct a grid of points in image A, transform it using the ground truth and neural network estimated transformations and with parameters and , respectively, and measure the discrepancy between the two transformed grids by summing the squared distances between the corresponding grid points:

where is the uniform grid used, and . We define the grid as having , that is to say, each coordinate belongs to a partition of in equally spaced subintervals of steps . Note that we construct the coordinate system such that the center of the image is at and that the width and height of the image are equal to , the bottom left and top right corners have coordinates and , respectively.

The gradient of the loss function with respect to the transformation parameters, needed to perform backpropagation in order to learn network weights, can be computed easily if the location of the transformed grid points is differentiable with respect to . This is commonly the case, for example, when is an affine transformation, is linear in parameters and therefore the loss can be differentiated in a straightforward manner.

4.2Training from /media/arxiv_projects/37807/synthetic transformations

Our training procedure requires fully supervised training data consisting of image pairs and a known geometric relation. Training CNNs usually requires a lot of data, and no public datasets exist that contain many image pairs annotated with their geometric transformation. Therefore, we opt for training from /media/arxiv_projects/37807/synthetically generated data, which gives us the flexibility to gather as many training examples as needed, for any 2-D geometric transformation of interest. We generate each training pair , by sampling from a public image dataset, and generating by applying a random transformation to . More precisely, is created from the central crop of the original image, while is created by transforming the original image with added symmetrical padding in order to avoid border artifacts; the procedure is shown in Figure 6.

Figure 6: Synthetic image generation. Symmetric padding is added to the original image to enlarge the sampling region, its central crop is used as image A, and image B is created by performing a randomly sampled transformation \mathcal{T}_{\theta_{GT}}.
Figure 6: Synthetic image generation. Symmetric padding is added to the original image to enlarge the sampling region, its central crop is used as image A, and image B is created by performing a randomly sampled transformation .

5Experimental results

In this section we describe our datasets, give implementation details, and compare our method to baselines and the state-of-the-art. We also provide further insights into the components of our architecture.

5.1Evaluation dataset and performance measure

Quantitative evaluation of our method is performed on the Proposal Flow dataset of Ham [23]. The dataset contains 900 image pairs depicting different instances of the same class, such as ducks and cars, but with large intra-class variations, the cars are often of different make, or the ducks can be of different subspecies. Furthermore, the images contain significant background clutter, as can be seen in Fig.  ?. The task is to predict the locations of predefined keypoints from image A in image B. We do so by estimating a geometric transformation that warps image A into image B, and applying the same transformation to the keypoint locations. We follow the standard evaluation metric used for this benchmark, the average probability of correct keypoint (PCK) [53], being the proportion of keypoints that are correctly matched. A keypoint is considered to be matched correctly if its predicted location is within a distance of of the target keypoint position, where and and are the height and width of the object bounding box, respectively.

5.2Training data

Two different training datasets for the affine and thin-plate spline stages, dubbed StreetView-/media/arxiv_projects/37807/synth-aff and StreetView-/media/arxiv_projects/37807/synth-tps respectively, were generated by applying /media/arxiv_projects/37807/synthetic transformations to images from the Tokyo Time Machine dataset [4] which contains Google Street View images of Tokyo.

Each /media/arxiv_projects/37807/synthetically generated dataset contains 40k images, divided into 20k for training and 20k for validation. The ground truth transformation parameters were sampled independently from reasonable ranges, for the affine transformation we sample the relative scale change of up to , while for thin-plate spline we randomly jitter a grid of control points by independently translating each point by up to one quarter of the image size in all directions.

In addition, a second training dataset for the affine stage was generated, created from the training set of Pascal VOC 2011 [16] which we dubbed Pascal-/media/arxiv_projects/37807/synth-aff. In Sec.  ?, we compare the performance of networks trained with StreetView-/media/arxiv_projects/37807/synth-aff and Pascal-/media/arxiv_projects/37807/synth-aff and demonstrate the generalization capabilities of our approach.

5.3Implementation details

We use the MatConvNet library [51] and train the networks with stochastic gradient descent, with learning rate , momentum 0.9, no weight decay and batch size of 16. There is no need for jittering as instead of data augmentation we can simply generate more /media/arxiv_projects/37807/synthetic training data. Input images are resized to producing feature maps that are passed into the matching layer. The affine and thin-plate spline stages are trained independently with the StreetView-/media/arxiv_projects/37807/synth-aff and StreetView-/media/arxiv_projects/37807/synth-tps datasets, respectively. Both stages are trained until convergence which typically occurs after 10 epochs, and takes 12 hours on a single GPU. Our final method for estimating affine transformations uses an ensemble of two networks that independently regress the parameters, which are then averaged to produce the final affine estimate. The two networks were trained on different ranges of affine transformations. As in Figure 5, the estimated affine transformation is used to warp image A and pass it together with image B to a second network which estimates the thin-plate spline transformation. All training and evaluation code, as well as our trained networks, are online at [1].

5.4Comparison to state-of-the-art

We compare our method against SIFT Flow [37], Graph-matching kernels (GMK) [15], Deformable spatial pyramid matching (DSP) [31], DeepFlow [43], and all three variants of Proposal Flow (NAM, PHM, LOM) [23]. As shown in Table 1, our method outperforms all others and sets the new state-of-the-art on this data. The best competing methods are based on Proposal Flow and make use of object proposals, which enables them to guide the matching towards regions of images that contain objects. Their performance varies significantly with the choice of the object proposal method, illustrating the importance of this guided matching. On the contrary, our method does not use any guiding, but it still manages to outperform even the best Proposal Flow and object proposal combination.

Furthermore, we also compare to affine transformations estimated with RANSAC using the same descriptors as our method (VGG-16 pool4). The parameters of this baseline have been tuned extensively to obtain the best result by adjusting the thresholds for the second nearest neighbor test and by pruning proposal transformations which are outside of the range of likely transformations. Our affine estimator outperforms the RANSAC baseline on this task with 49% (ours) compared to 47% (RANSAC).

Table 1: Comparison to state-of-the-art and baselines. Matching quality on the Proposal Flow dataset measured in terms of PCK. The Proposal Flow methods have four different PCK values, one for each of the four employed region proposal methods. All the numbers apart from ours and RANSAC are taken from .
Methods PCK (%)
DeepFlow 20
GMK 27
SIFT Flow 38
DSP 29
Proposal Flow NAM 53
Proposal Flow PHM 55
Proposal Flow LOM 56
RANSAC with our features (affine) 47
Ours (affine) 49
Ours (affine + thin-plate spline) 56
Ours (affine ensemble + thin-plate spline) 57

5.5Discussions and ablation studies

In this section we examine the importance of various components of our architecture. Apart from training on the StreetView-/media/arxiv_projects/37807/synth-aff dataset, we also train on Pascal-/media/arxiv_projects/37807/synth-aff which contains images that are more similar in nature to the images in the Proposal Flow benchmark. The results of these ablation studies are summarized in Table 2.

Table 2: Ablation studies. Matching quality on the Proposal Flow dataset measured in terms of PCK. All methods use the same features (VGG-16 cropped at pool4). The networks were trained on the StreetView-/media/arxiv_projects/37807/synth-aff and Pascal-/media/arxiv_projects/37807/synth-aff datasets. For these experiments, only the affine transformation is estimated.
Methods StreetView-/media/arxiv_projects/37807/synth-aff Pascal-/media/arxiv_projects/37807/synth-aff
Concatenation 26 29
Subtraction 18 21
Ours without normalization 44
Ours 49 45
Figure 7: Filter visualization. Some convolutional filters from the first layer of the regressor, acting on the tentative correspondence map, show preferences to spatially co-located features that transform consistently to the other image, thus learning to perform the local neighborhood consensus criterion often used in classical feature matching. Refer to the text for more details on the visualization.
Figure 7: Filter visualization. Some convolutional filters from the first layer of the regressor, acting on the tentative correspondence map, show preferences to spatially co-located features that transform consistently to the other image, thus learning to perform the local neighborhood consensus criterion often used in classical feature matching. Refer to the text for more details on the visualization.

Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix .
Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix .
Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix . Qualitative results on the Proposal Flow dataset. Each row shows one test example from the Proposal Flow dataset. Ground truth matching keypoints, only used for alignment evaluation, are depicted as crosses and circles for images A and B, respectively. Keypoints of same color are supposed to match each other after image A is aligned to image B. To illustrate the matching error, we also overlay keypoints of B onto different alignments of A so that lines that connect matching keypoints indicate the keypoint position error vector. Our method manages to roughly align the images with an affine transformation (column 2), and then perform finer alignment using thin-plate spline (TPS, column 3). It successfully handles background clutter, translations, rotations, and large changes in appearance and scale, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix .
Image A Aligned A (affine) Aligned A (affine+TPS) Image B

Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map).
Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map). Qualitative results on the Tokyo Time Machine dataset. Each row shows a pair of images from the Tokyo Time Machine dataset, and our alignment along with a difference map, highlighting absolute differences between aligned images in the descriptor space. Our method successfully aligns image A to image B despite of viewpoint and scene changes (highlighted in the difference map).
(a) Image A (b) Image B (c) Aligned image A (d) Overlay of (b) and (c) (e) Difference map

Replacing our correlation-based matching layer with feature concatenation or subtraction, as proposed in [14] and [30], respectively, incurs a large performance drop. The behavior is expected as we designed the matching layer to only keep information on pairwise descriptor similarities rather than the descriptors themselves, as is good practice in classical geometry estimation methods, while concatenation and subtraction do not follow this principle.

As seen in Table 2, our method is relatively unaffected by the choice of training data as its performance is similar regardless whether it was trained with StreetView or Pascal images. We also attribute this to the design choice of operating on pairwise descriptor similarities rather than the raw descriptors.

Table 2 also shows the importance of the correlation map normalization step, where the normalization improves results from 44% to 49%. The step mimics the second nearest neighbor test used in classical feature matching [40], as discussed in Section 3.2. Note that [18] also uses a correlation layer, but they do not normalize the map in any way, which is clearly suboptimal.

We examine filters from the first convolutional layer of the regressor, which operate directly on the output of the matching layer, the tentative correspondence map. Recall that each spatial location in the correspondence map (see Figure 3, in green) contains all similarity scores between that feature in image B and all features in image A. Thus, each single 1-D slice through the weights of one convolutional filter at a particular spatial location can be visualized as an image, showing filter’s preferences to features in image B that match to specific locations in image A. For example, if the central slice of a filter contains all zeros apart from a peak at the top-left corner, this filter responds positively to features in image B that match to the top-left of image A. Similarly, if many spatial locations of the filter produce similar visualizations, then this filter is highly sensitive to spatially co-located features in image B that all match to the top-left of image A. For visualization, we pick the peaks from all slices of filter weights and average them together to produce a single image. Several filters shown in Figure 7 confirm our hypothesis that this layer has learned to mimic local neighborhood consensus as some filters respond strongly to spatially co-located features in image B that match to spatially consistent locations in image A. Furthermore, it can be observed that the size of the preferred spatial neighborhood varies across filters, thus showing that the filters are discriminative of the scale change.

5.6Qualitative results

Fig.  ? illustrates the effectiveness of our method in category-level matching, where challenging pairs of images from the Proposal Flow dataset [23], containing large intra-class variations, are aligned correctly. The method is able to robustly, in the presence of clutter, estimate large translations, rotations, scale changes, as well as non-rigid transformations and some perspective changes. Further examples are shown in appendix ?.

Fig.  ? shows the quality of instance-level matching, where different images of the same scene are aligned correctly. The images are taken from the Tokyo Time Machine dataset [4] and are captured at different points in time which are months or years apart. Note that, by automatically highlighting the differences (in the feature space) between the aligned images, it is possible to detect changes in the scene, such as occlusions, changes in vegetation, or structural differences new buildings being built.

6Conclusions

We have described a network architecture for geometric matching fully trainable from /media/arxiv_projects/37807/synthetic imagery without the need for manual annotations. Thanks to our matching layer, the network generalizes well to never seen before imagery, reaching state-of-the-art results on the challenging Proposal Flow dataset for category-level matching. This opens-up the possibility of applying our architecture to other difficult correspondence problems such as matching across large changes in illumination (day/night) [4] or depiction style [5].

This work has been partly supported by ERC grant LEAP (no. 336845), ANR project Semapolis (ANR-13-CORD-0003), the Inria CityLab IPL, CIFAR Learning in Machines Brains program and ESIF, OP Research, development and education Project IMPACT No. CZ.

References

  1. http://www.di.ens.fr/willow/research/cnngeometric/.
    Project webpage (code/networks).
  2. Building Rome in a day.
    S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and R. Szeliski. In Proc. ICCV, 2009.
  3. Learning to match aerial images with deep attentive architectures.
    H. Altwaijry, E. Trulls, J. Hays, P. Fua, and S. Belongie. In Proc. CVPR, 2016.
  4. NetVLAD: CNN architecture for weakly supervised place recognition.
    R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. In Proc. CVPR, 2016.
  5. Painting-to-3D model alignment via discriminative visual elements.
    M. Aubry, B. Russell, and J. Sivic. ACM Transactions on Graphics, 2013.
  6. Factors of transferability from a generic ConvNet representation.
    H. Azizpour, A. Razavian, J. Sullivan, A. Maki, and S. Carlsson. arXiv preprint arXiv:1406.5774, 2014.
  7. Aggregating local deep features for image retrieval.
    A. Babenko and V. Lempitsky. In Proc. ICCV, 2015.
  8. PN-Net: Conjoined triple deep network for learning local image descriptors.
    V. Balntas, E. Johns, L. Tang, and K. Mikolajczyk. arXiv preprint arXiv:1601.05030, 2016.
  9. Surf: Speeded up robust features.
    H. Bay, T. Tuytelaars, and L. Van Gool. In Proc. ECCV, 2006.
  10. Shape matching and object recognition using low distortion correspondence.
    A. Berg, T. Berg, and J. Malik. In Proc. CVPR, 2005.
  11. Principal warps: Thin-plate splines and the decomposition of deformations.
    F. L. Bookstein. IEEE PAMI, 1989.
  12. Histogram of Oriented Gradients for Human Detection.
    N. Dalal and B. Triggs. In Proc. CVPR, 2005.
  13. ImageNet: A large-scale hierarchical image database.
    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. In Proc. CVPR, 2009.
  14. Deep image homography estimation.
    D. DeTone, T. Malisiewicz, and A. Rabinovich. arXiv preprint arXiv:1606.03798, 2016.
  15. A graph-matching kernel for object categorization.
    O. Duchenne, A. Joulin, and J. Ponce. In Proc. ICCV, 2011.
  16. The PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results.
    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. http://www.pascal-network.org/challenges/VOC/voc2011/workshop/index.html.
  17. One-shot learning of object categories.
    L. Fei-Fei, R. Fergus, and P. Perona. IEEE PAMI, 2006.
  18. FlowNet: Learning optical flow with convolutional networks.
    P. Fischer, A. Dosovitskiy, E. Ilg, P. Häusser, C. Hazirbaş, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. In Proc. ICCV, 2015.
  19. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.
    M. A. Fischler and R. C. Bolles. Comm. ACM, 1981.
  20. Computer vision: a modern approach.
    D. A. Forsyth and J. Ponce. Prentice Hall Professional Technical Reference, 2002.
  21. Multi-scale orderless pooling of deep convolutional activation features.
    Y. Gong, L. Wang, R. Guo, and S. Lazebnik. In Proc. ECCV, 2014.
  22. Non-rigid dense correspondence with applications for image enhancement.
    Y. HaCohen, E. Shechtman, D. B. Goldman, and D. Lischinski. Proc. ACM SIGGRAPH, 2011.
  23. Proposal Flow.
    B. Ham, M. Cho, C. Schmid, and J. Ponce. In Proc. CVPR, 2016.
  24. MatchNet: Unifying feature and metric learning for patch-based matching.
    X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg. In Proc. CVPR, 2015.
  25. A combined corner and edge detector.
    C. Harris and M. Stephens. In Alvey vision conference, 1988.
  26. Multiple view geometry in computer vision.
    R. Hartley and A. Zisserman. Cambridge university press, 2003.
  27. Batch Normalization: Accelerating deep network training by reducing internal covariate shift.
    S. Ioffe and C. Szegedy. In Proc. ICML, 2015.
  28. “Spatial Transformer Networks”.
    M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. In NIPS, 2015.
  29. Learned local descriptors for recognition and matching.
    M. Jahrer, M. Grabner, and H. Bischof. In Computer Vision Winter Workshop, 2008.
  30. WarpNet: Weakly supervised matching for single-view reconstruction.
    A. Kanazawa, D. W. Jacobs, and M. Chandraker. In Proc. CVPR, 2016.
  31. Deformable spatial pyramid matching for fast dense correspondences.
    J. Kim, C. Liu, F. Sha, and K. Grauman. In Proc. CVPR, 2013.
  32. Deformable spatial pyramid matching for fast dense correspondences.
    J. Kim, C. Liu, F. Sha, and K. Grauman. IEEE PAMI, 2013.
  33. ImageNet classification with deep convolutional neural networks.
    A. Krizhevsky, I. Sutskever, and G. E. Hinton. In NIPS, 2012.
  34. Object recognition by affine invariant matching.
    Y. Lamdan, J. T. Schwartz, and H. J. Wolfson. In Proc. CVPR, 1988.
  35. Data driven image models through continuous joint alignment.
    E. G. Learned-Miller. IEEE PAMI, 2006.
  36. Robust object detection with interleaved categorization and segmentation.
    B. Leibe, A. Leonardis, and B. Schiele. IJCV, 2008.
  37. SIFT Flow: Dense correspondence across scenes and its applications.
    C. Liu, J. Yuen, and A. Torralba. IEEE PAMI, 2011.
  38. Do convnets learn correspondence?
    J. L. Long, N. Zhang, and T. Darrell. In NIPS, 2014.
  39. Object recognition from local scale-invariant features.
    D. G. Lowe. In Proc. ICCV, 1999.
  40. Distinctive image features from scale-invariant keypoints.
    D. G. Lowe. IJCV, 2004.
  41. An affine invariant interest point detector.
    K. Mikolajczyk and C. Schmid. In Proc. ECCV, 2002.
  42. Object retrieval with large vocabularies and fast spatial matching.
    J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. In Proc. CVPR, 2007.
  43. DeepMatching: Hierarchical deformable dense matching.
    J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid. IJCV, 2015.
  44. Unsupervised joint object discovery and segmentation in internet images.
    M. Rubinstein, A. Joulin, J. Kopf, and C. Liu. In Proc. CVPR, 2013.
  45. Local grayvalue invariants for image retrieval.
    C. Schmid and R. Mohr. IEEE PAMI, 1997.
  46. Unsupervised visual alignment with similarity graphs.
    F. Shokrollahi Yancheshmeh, K. Chen, and J.-K. Kamarainen. In Proc. CVPR, 2015.
  47. Discriminative learning of deep convolutional feature point descriptors.
    E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer. In Proc. ICCV, 2015.
  48. Very deep convolutional networks for large-scale image recognition.
    K. Simonyan and A. Zisserman. In Proc. ICLR, 2015.
  49. Video Google: A text retrieval approach to object matching in videos.
    J. Sivic and A. Zisserman. In Proc. ICCV, 2003.
  50. Fully-trainable deep matching.
    J. Thewlis, S. Zheng, P. Torr, and A. Vedaldi. In Proc. BMVC., 2016.
  51. MatConvNetConvolutional neural networks for MATLAB.
    A. Vedaldi and K. Lenc. In Proc. ACMM, 2015.
  52. DeepFlow: Large displacement optical flow with deep matching.
    P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. In Proc. ICCV, 2013.
  53. Articulated human detection with flexible mixtures of parts.
    Y. Yang and D. Ramanan. IEEE PAMI, 2013.
  54. Learning to compare image patches via convolutional neural networks.
    S. Zagoruyko and N. Komodakis. In Proc. CVPR, 2015.
  55. “FlowWeb: Joint image set alignment by weaving consistent, pixel-wise correspondences”.
    T. Zhou, Y. J. Lee, S. X. Yu, and A. A. Efros. In Proc. CVPR, 2015.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
22670
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description