Separable Four Points Fundamental Matrix
We present a novel approach for RANSAC-based computation of the fundamental matrix based on epipolar homography decomposition. We analyze the geometrical meaning of the decomposition-based representation and show that it directly induces a consecutive sampling strategy of two independent sets of correspondences. We show that our method guarantees a minimal number of evaluated hypotheses with respect to current minimal approaches, on the condition that there are four correspondences on an image line. We validate our approach on real-world image pairs, providing fast and accurate results.
One of the basic building blocks in computer vision is the computation of epipolar geometry given a set of putative image point correspondences. Often, such correspondences include mismatches, therefore a robust estimation method needs to be carried out. The most common method used is RANSAC . RANSAC-based computation of the fundamental matrix iteratively samples a minimal set of putative correspondences and hypothesizes the fundamental matrix parameters. The goodness-of-fit of each hypothesis is evaluated with respect to the whole set of the putative correspondences. The number of evaluations at each iteration is between one to three, depending on the number of possible solutions. This process is repeated until a predefined number of iterations is exceeded. One of its key limitations is that in the presence of a considerable amount of mismatched points’ correspondences, a large number of hypotheses’ evaluations is needed in order to obtain a reliable model. The number of the required hypotheses is directly related to the number of points needed to be sampled in order to hypothesize the fundamental matrix parameters. From a geometrical point of view, the minimal sample size is at least seven . Minimal sample approaches have been proposed requiring only six  or five points’ correspondences [24, 3], with additional knowledge regarding the parameters or the scene.
We present a method that can reduce the minimal number of RANSAC hypotheses’ evaluations beyond the current known minimum. Our approach, similar to approaches that require less than seven correspondences, assumes additional knowledge. We require that a single line segment exists in the image with four correspondences and show experimentally that such a configuration is common. Given such a configuration, we show that our approach guarantees a minimal number of hypotheses’ evaluations. In cases where such a configuration does not exist, standard RANSAC can be applied with overall small overhead. Hereafter, when referring to RANSAC iterations, we will refer to the actual number of hypotheses’ evaluations which depends upon the number of solutions per hypothesis.
We compute the fundamental matrix in two steps. In the first step, we sample three points on a line segment. We compute the epipolar homography and validate its goodness of fit using at least one additional point, without the full computation of the fundamental matrix. In the second step, the recovered matches are fixed and the remaining points’ matches are recovered by sampling, without any restriction. Our approach is based on the computation of the epipolar homography. Existing methods for the computation of the epipolar homography [13, 23, 5, 10, 12] require the knowledge of (at least) three corresponding epipolar lines and, as a consequence, are rarely used in practice. We show how to compute the epipolar homography without a-priori knowledge of epipolar lines. Figure 1 shows the recovered four matches in the first step of our approach and the extracted epipolar lines based on our epipolar homography computation.
This paper therefore contributes by presenting: (a) a reformulation of the fundamental matrix based on epipolar homography decomposition and an analysis of its geometrical meaning, (b) a novel two steps RANSAC-based approach for the computation of the fundamental matrix that can markedly reduce the required number of hypotheses’ evaluations and (c) an efficient implementation and a validation of our approach on real-world image pairs showing that it is accurate and applicable.
2 Related Work
The most common approaches for the computation of the fundamental matrix are the seven or eight point algorithms . The eight-point algorithm  which was adapted for the fundamental matrix, was made practical by . It is based on the normalization of the points’ correspondences, and computes the parameters based on the Direct Linear Transform (DLT) by enforcing the rank 2 constraint . The seven points algorithm relaxes the eight points requirement by an additional zero determinant constraint of the matrix, resulting in a cubic equation with one or three real solutions.
Various minimal methods assume additional knowledge in order to reduce the required number of correspondences. If the camera parameters are known, the five-points algorithm  can be used. Recently, Barath  proposed an approach for the estimation of the fundamental matrix based on five correspondences, in cases where three co-planar point correspondences and the rotation of features are known. Ben-Artzi et al.  showed that in case the intensity of the epipolar lines are similar across views, two corresponding points are sufficient for the computation of matching epipolar lines, and can be used for recovery of the fundamental matrix.
Methods attempting to empirically reduce the number of RANSAC iterations have also been introduced [1, 6, 19]. Unlike such methods, our approach, as well as other minimal methods, guarantees to reduce the number of required samples and this number can be calculated in advance in the same way as in standard RANSAC. In addition, our approach has a true geometrical meaning (See Sec. 3), whereas no geometrical analysis is possible in such methods.
The fundamental matrix can be computed based on the epipolar homography [12, 8]. These methods require the knowledge of epipolar line correspondences. Sinha  introduced an approach for camera calibration based on the computation of the epipolar homography and epipoles hypothesizing. His approach was later improved by , who directly recovered the epipolar lines required for the computation of the epipolar homography by using the motion-barcode descriptor. Similar approaches were introduced by [13, 10] for the computation of the epipolar homography by directly estimating the epipolar lines. However, all of the above methods are only applicable to videos of dynamic scenes. Wurfl et. al.  presented an approach for estimating the fundamental matrix transmission imaging, based on the epipolar homography formulation. They do not require explicit correspondences but rather sampling of all edge pixels of the image. The computation of the relative pose by general line homography based on line segments was demonstrated by  but they required the matching points to be the projection of existing 3D lines in the scene and required at least two such lines.
3 Theoretical Background
Epipolar-Lines Based Parametrization. Let be corresponding epipolar lines, in the first and second image, respectively. Let be a line that intersects in the point and does not include the epipole. It follows that = , where is the cross product. Since is on , it follows that . Denote as the skew symmetric matrix associated with the cross product, we have the following mapping:
where is the epipolar homography. The seven degrees of freedom of the fundamental matrix is constructed by the four degrees of freedom of the two epipoles and the three degrees of freedom of the epipolar homography . The epipolar homography is obtained by three corresponding epipolar lines.
Image-Points’ Based Parametrization. Consider the set of epipolar lines passing through the epipole which is denoted as the pencil of epipolar lines. As shown in Fig. 2, the corresponding epipolar lines across images define a plane which intersects the two retinal planes with its axis as the baseline. When the corresponding epipolar lines rotate around the epipoles, their corresponding planes rotate around the baseline, defining the pencil of epipolar planes. The two corresponding pencils of epipolar lines are related by the epipolar homography based on the pencil of epipolar planes . Let be corresponding epipolar lines in the first and second image respectively. Let be arbitrary lines in the first and second image, such that the epipoles are not on the lines. The epipolar homography maps between based on their intersection points with as follows. Let and , denoted as control points. The intersection points of with are and , where is the cross product and using homogeneous coordinates of the control points from now on. Using the following identity
for , the intersection points can be written as a linear combination of the control points as and , where
. The epipolar homography is given by:
Rewriting the corresponding epipolar lines as the cross product of the epipoles and points on the epipolar lines, and expanding the intersection points’ coefficients, we have the following parametrization of the fundamental matrix:
Observation 1. The epipolar homography maps between the corresponding points based on their representation as a linear combination of their respective control points. It can be computed without knowledge of the corresponding epipolar lines. Given the line segments of the points, we select (fixed) control points on the lines and represent the points as their linear combination. The points-based parametrization of the pencil of epipolar lines by their intersection with the lines is shown in Fig. 3.
Observation 2. The required number of RANSAC iterations for the full computation of the fundamental matrix can be greatly reduced. First, we recover which is a three parameters model without the need for the full computation of the fundamental matrix. We can then recover four additional arbitrary correspondences by relying on the recovered . Let be the required number of RANSAC iterations for a model with parameters, outlier rate and success probability of . The required number of RANSAC iterations is now instead of . Fig. 4 shows the required number of RANSAC iterations for the computation of the fundamental matrix by our approach and existing minimal methods, as a function of the outlier rate. Our approach can be implemented based on the seven-point or eight-point algorithm; both are presented in the figure. The number of RANSAC iterations is computed assuming one solution per sample and no additional overhead for any of the methods. In Sec. 6, we present the expected number of RANSAC iterations based on the actual number of solutions per sample in the tested datasets.
Observation 3. The lines with the corresponding points are arbitrary line segments in the images. Figure 5 illustrates the mapping of points in the first image to lines in the second image, based on the parametrization of the fundamental matrix using the points-based epipolar homography (Eq. 2), given .
4 Line Segments with Corresponding Points
In this section we describe how to find the lines across images with the maximum number of putative correspondences. For each image, our approach does not require knowledge of the points and lines in the other image, which allows parallel implementation. The key idea is to use Hough transform  to detect the lines independently in each image and update a joint accumulator which can be queried only after the process is completed. Fig. 6 illustrates the matching process of corresponding lines across images.
Let denote a pair of images. Let denote the set of ordered pairs of unique putative corresponding image points where are image points in , respectively. Let , . We create binary images such that for each pixel in , if otherwise , and similarly for . We use Hough transform  to extract the ordered set of lines in , respectively. We consider only lines with at least four correspondences. This requirement is due to the robust estimation process of the homography (See Sec. 5), where we sample three points and need at least one additional point to estimate the quality of the hypothesized homography.
We define a multidimensional array, an accumulator of size , where and are the number of lines in and , respectively, and is the number of putative corresponding points. The accumulator is initialized to zero. Let be the indices of nearby lines in for a point in image ,
where is the point to line distance and is the constant representing the required distance between the lines and points. For each point in the first image, we increment the accumulator according to its nearby lines in the first image and all existing lines in the second image, s.t.
For each point in the second image, is defined similarly and the accumulator entries are incremented according to
The pair of matching lines with the maximum number of putative correspondences are given by where
In case there is more than a single line we randomly select one.
5 Separable Four Points Fundamental Matrix
Given a set of putative correspondences , we compute the fundamental matrix by two steps. Our computation is based on RANSAC. We assume inlier ratio , epipolar homography inlier threshold and fundamental matrix inlier threshold .
Step One. Based on Sec. 4 , we recover , which are the corresponding image lines with the maximum number of putative corresponding points . We select fixed control points for each line and . The points are selected on the intersection of the lines and the boundaries of the images. For each pair of putative corresponding points on the lines, we compute their representation by the control points with the two possible orientations and use the second one only if the first fails. We use RANSAC to recover the epipolar homography. We iteratively sample three putative corresponding points, compute the epipolar homography, transfer each point on the line to the corresponding line, and count the number of inliers based on the Euclidean distance and the threshold . The number of required iterations for the three parameter model is often small. For example, for an outlier rate of 60%, we need only 71 iterations to recover the correct model with a confidence of 99%. Note that although we sample three points, we require at least four corresponding points on line segment; the additional points are used for validation of the hypothesized epipolar homography.
Step Two. Given the three points selected, we iteratively sample four additional points and compute the fundamental matrix based on the seven points algorithm. The number of iterations is computed based on with respect to four parameters. We count the number of inliers based on with respect to the symmetric epipolar distance . For the same case as above of an outlier rate of 60%, for the four parameters’ model we need only 178 iterations.
The above refers to the implementation of our approach based on the seven points algorithm. For implementation based on the eight points algorithm, we sample five points instead of four in the second step.
We conduct experiments on the following real world datasets: (1) The Strecha datasets  which include ground truth cameras and reconstruction, (2) The Tower of London sequence from the Flickr dataset , where the images were downloaded from Flickr based on geotag. It includes images captured by the community at random camera positions with a large variety of urban outdoor scenes, and (3) The Family sequence from the Tanks and Temples dataset , which includes wide baseline camera poses, with medium sized images captured by a hand-held camera in an urban yard scene.
For the last two datasets, we sampled 2000 image pairs with at least 20 matches with a symmetric epipolar distance of less than or equal to 1 pixel with respect to the ground truth, and used them for the evaluation. We show that our approach is able to successfully compute the fundamental matrices for these real-world diverse sets of image pairs, leading to an efficient estimation process. Table 1 shows the properties of the datasets.
|Tanks and Temples||2000||COLMAP|
6.2 Matching Co-Linear Points
In this subsection, we discuss the matching process of corresponding lines with at least four points, based on the Strecha dataset.
How Many Lines? On average, the Hough transform recovers 286.4 independent lines in each image. Out of all possible pairs of matched lines, the average number of lines with corresponding points across a pair of images is 3354.91, including lines with outliers. 45.8% (1538.84) of the matched lines are lines with at least four correspondences, where a corresponding point might be either inlier or outlier.
Quality of Lines. Overall, we would like to match lines with a high number of inliers and a low homography error. The average number of points, including outliers, on matched lines is 7.62. The average number of inliers on corresponding lines is 5.83, for a homography threshold of 1. Fig. 7 shows the probability of having a given number of inliers in matched lines, for such a homography error. Fig. 8 presents the probability for a given homography error on the matched lines. The homography error of 97% of the inliers is below and 99% are below 0.5, which is used as our default threshold.
Scene Structure. Our approach often successfully matches lines in both near and far scenes and low and high inlier rates. Fig. 9 shows examples from Tower of London dataset of matched lines and corresponding points recovered by our approach, with inlier ratios of and , for various scenes. For the first and second row from the top, there are 27 and 25 inliers out of 148 and 89 putative correspondences, respectively. For the last row, there are 498 inliers out of 582 putative correspondences.
Missing Inliers Relative to RANSAC, our approach failed to recover at least 20 inliers across images in 4.9% of the cases. Qualitatively, these cases occur when there is only a small number of inliers, both in the images and on the recovered lines. For the failure cases, the average number of inliers is 26.8 and the maximum is 34. Quantitative analysis indicates that in such cases the best possible matching of corresponding lines across images contains wrong or inaccurate matches and the homography could not be recovered. Fig. 10 shows such an example. The overall percentages of failure cases over all datasets are presented in Table 5.
RANSAC with the standard eight points algorithm, denoted as RANSAC.
LMEDS with the standard eight points algorithm, denoted as LMEDS.
RANSAC with our minimal sampling solver, using the seven points algorithm based on two samples of three and four points, denoted as Ours-RANSAC-4.
RANSAC with our minimal sampling solver, using the eight points algorithm based on two samples of three and five points, denoted as Ours-RANSAC-5.
LMEDS with our minimal sampling solver, using the seven points algorithm based on two samples of three and four points, denoted as Ours-LMEDS-4.
LMEDS with our minimal sampling solver, using the eight points algorithm based on two samples of three and five points, denoted as Ours-LMEDS-5.
Methodology. For the baseline methods we performed a grid-search over hyperparameters. For sequences with no ground truth fundamental matrices, we reconstructed the sequences using COLMAP . COLMAP is a general purpose Structure-from-Motion (SfM) pipeline which globally infers the 3D structure, leading to accurate results. We obtained the camera poses and fundamental matrices, and used them as the ground truth.
For all sequences, we used the SIFT  descriptor to extract and match putative correspondences between all pairs of images. We used the ground truth fundamental matrices to evaluate the symmetric epipolar distance of the putative matching. Each image pair with less than 20 matches within a symmetric epipolar distance of one is discarded.
Metrics. We report the average symmetric epipolar distance resulting from our approach and the baselines. Unless otherwise stated, we used a threshold of 3 pixels in our approach as the inlier threshold and as the epipolar homography threshold. We use a fixed homographyâs threshold over all the datasets, demonstrating its robustness. We report the percentage of inliers found by each method, and report the F-measure, where corresponding points with symmetric epipolar distance less than 1 pixel with respect to the ground truth fundamental matrix is considered as positive.
Strecha. The dataset contains eight multi-view collections of high-resolution images (), provided with ground truth camera poses. For the computation of matching lines with putative corresponding points across images, we sampled putative correspondences, and used these lines with respect to all existing points’ correspondences. We used SIFT ratio-test of . The binary images used for matching the lines (Sec. 4) are of a lower resolution, as suggested in [26, 27], and the points coordinates are resized accordingly. Unless noted otherwise, the width resolution of was used for the line matching phase with the original aspect ratio. The reported results are with respect to the original resolutions. The sequences are of images with varying distances between focal points with a total of 569 valid image pairs. As suggested in , we used the five-points’ solver of  within the GC-RANSAC  estimator. Table. 2 presents our results. In both %-inliers and F-Score metrics our RANSAC-based eight points approach outperforms the baselines.
Flickr. The dataset obtained by downloading images of specific scenes from Flickr represents the most challenging case. The images captured random scenes at different times, with various cameras from very different positions. We use the Tower of London sequence for comparison. We reconstruct the sequence using COLMAP and use the reconstruction as ground truth. The results are presented in Table 3, where it can be seen that our RANSAC-based eight points approach clearly outperforms all other methods.
Tanks and Temples. The dataset includes medium-resolution images (). For comparison, we used the Family sequence. As ground truth, we use the COLMAP Reconstruction. As shown in Table 4, our RANSAC-based eight points approach recovered the highest number of inliers, with mean error similar to . Both outperform the baselines.
We measured the number of RANSAC iterations required by each method and the number of failure cases. For the total number of RANSAC iterations, we also consider all validation iterations. For methods which are based on the seven points algorithm, both ours and the baseline, there might be more iterations than the theoretical number of required samples. A failure is considered if the solver has not been able to accurately recover at least 20 matches, in case there are indeed such ground truth matches.
The solver of  is combined within another robust estimator framework, the GC-RANSAC . We execute only the solver itself with the optimal number of iterations as an initialization. Our approach starts by finding the matching lines with the maximum number of putative correspondences (Sec. 5). Based on our experiments the runtime of our lines’ matching step is equivalent to 71 RANSAC iterations and the number of solutions per sample for the 7-point algorithm is 2.43. To reduce the additional overhead associated with our matching step we use instead the standard eight points’ RANSAC for inlier ratio less than . For such inlier ratio the 8-point algorithm requires only one hypothesis’ evaluation per sample, with overall - iterations instead of - for the 7-point algorithm. Fig. 11 presents the required number of iterations by the different approaches based on the actual number of solutions per sample including the additional iterations required by our matching lines phase.
Table 5 shows the percentage of cases that failed for each approach and the average number of iterations, over all the three datasets, the Strecha, Flickr and Tanks and Temples.
We presented a novel approach for the computation of the fundamental matrix based on epipolar homography decomposition. Our approach can reduce, both theoretically and practically, the number of required iterations to a new minimal number. The standard RANSAC procedure can be incorporated into our approach, providing an overall robust and efficient solution which is well suited for Structure from Motion (SfM) pipelines.
- (2019) MAGSAC: marginalizing sample consensus. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10197–10205. Cited by: §2.
- (2018) Graph-cut ransac. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6733–6741. Cited by: 3rd item, §6.4, §6.5.
- (2018) Five-point fundamental matrix estimation for uncalibrated cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 235–243. Cited by: §1, §2, Figure 4, 3rd item, §6.4, §6.4, §6.5.
- (2016) Epipolar geometry based on line similarity. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 1864–1869. Cited by: §2.
- (2016) Camera calibration from dynamic silhouettes using motion barcodes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4095–4103. Cited by: §1, §2.
- (2005) Matching with prosac-progressive sample consensus. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), Vol. 1, pp. 220–226. Cited by: §2.
- (1971) Use of the hough transformation to detect lines and curves in pictures. Technical report Sri International Menlo Park Ca Artificial Intelligence Center. Cited by: §4, §4.
- (2001) The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications. MIT press. Cited by: §2, §3.
- (1981-06) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24 (6), pp. 381–395. External Links: Cited by: §1, §6.3.
- (2018) An epipolar line from a single pixel. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 983–991. Cited by: §1, §2.
- (1997) In defense of the eight-point algorithm. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19 (6), pp. 580–593. Cited by: §2.
- (2003) Multiple view geometry in computer vision. Cambridge university press. Cited by: §1, §1, §2, §2, §3, §5.
- (2016) Fundamental matrices from moving objects using line motion barcodes. In European Conference on Computer Vision, pp. 220–228. Cited by: §1, §2.
- (2017) Tanks and temples: benchmarking large-scale scene reconstruction. ACM Transactions on Graphics 36 (4). Cited by: §6.1.
- (1987) A computer algorithm for reconstructing a scene from two projections. Readings in Computer Vision: Issues, Problems, Principles, and Paradigms, MA Fischler and O. Firschein, eds, pp. 61–62. Cited by: §2.
- (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60 (2), pp. 91–110. Cited by: §6.3.
- (1996) The fundamental matrix: theory, algorithms, and stability analysis. International journal of computer vision 17 (1), pp. 43–75. Cited by: §2.
- (2004) An efficient solution to the five-point relative pose problem. IEEE transactions on pattern analysis and machine intelligence 26 (6), pp. 0756–777. Cited by: §2.
- (2005) Preemptive ransac for live structure and motion estimation. Machine Vision and Applications 16 (5), pp. 321–329. Cited by: §2.
- (2010) 3D camera pose estimation using line correspondences and 1d homographies. In International Symposium on Visual Computing, pp. 41–52. Cited by: §2.
- (1984) Least median of squares regression. Journal of the American statistical association 79 (388), pp. 871–880. Cited by: §6.3.
- (2016) Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §6.3, §6.4.
- (2010) Camera network calibration and synchronization from silhouettes in archived video. International journal of computer vision 87 (3), pp. 266–283. Cited by: §1, §2.
- (2005) A minimal solution for relative pose with unknown focal length. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2, pp. 789–794. Cited by: §1.
- (2008) On benchmarking camera calibration and multi-view stereo for high resolution imagery. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. Cited by: §6.1.
- (2009) Daisy: an efficient dense descriptor applied to wide-baseline stereo. IEEE transactions on pattern analysis and machine intelligence 32 (5), pp. 815–830. Cited by: §6.4.
- (2013) Dense segmentation-aware descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2890–2897. Cited by: §6.4.
- (1997) Six-point synthetic method to estimate fundamental matrix. Science in China Series E: Technological Sciences 40 (2), pp. 199–205. Cited by: §1.
- (2014) Robust global translations with 1dsfm. In European Conference on Computer Vision, pp. 61–75. Cited by: §6.1.
- (2019) Estimating the fundamental matrix without point correspondences with application to transmission imaging. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1072–1081. Cited by: §2.