Multi-Class Model Fitting by Energy Minimization and Mode-Seeking
We propose a general formulation, called Multi-X, for multi-class multi-instance model fitting – the problem of interpreting the input data as a mixture of noisy observations originating from multiple instances of multiple classes. We extend the commonly used -expansion-based technique with a new move in the label space. The move replaces a set of labels with the corresponding density mode in the model parameter domain, thus achieving fast and robust optimization. Key optimization parameters like the bandwidth of the mode seeking are set automatically within the algorithm. Considering that a group of outliers may form spatially coherent structures in the data, we propose a cross-validation-based technique removing statistically insignificant instances. Multi-X outperforms significantly the state-of-the-art on publicly available datasets for diverse problems: multiple plane and rigid motion detection; motion segmentation; simultaneous plane and cylinder fitting; circle and line fitting.
In multi-class fitting, the input data is interpreted as a mixture of noisy observations originating from multiple instances of multiple model classes, e.g. lines and circles in 2D edge maps, planes and cylinders in 3D data, multiple homographies or fundamental matrices from correspondences from a non-rigid scene (see Fig. ?). Robustness is achieved by considering assignment to an outlier class.
Multi-model fitting has been studied since the early sixties, the Hough-transform  being the first popular method for extracting multiple instances of a single class . A widely used approach for finding a single instance is RANSAC  which alternates two steps: the generation of instance hypotheses and their validation. However, extending RANSAC to the multi-instance case has had limited success. Sequential RANSAC detects instance one after another in a greedy manner, removing their inliers . In this approach, data points are assigned to the first instance, typically the one with the largest support for which they cannot be deemed outliers, rather than to the best instance. MultiRANSAC  forms compound hypothesis about instances. Besides requiring the number of the instances to be known a priori, the approach increases the size of the minimum sample and thus the number of hypotheses that have to be validated.
Most recent approaches  focus on the single class case: finding multiple instances of the same model class. A popular group of methods  adopts a two step process: initialization by RANSAC-like instance generation followed by a point-to-instance assignment optimization by energy minimization using graph labeling techniques . Another group of methods uses preference analysis, introduced by RHA , which is based on the distribution of residuals of individual data points with respect to the instances .
The multiple instance multiple class case considers fitting of instances that are not necessarily of the same class. This generalization has received much less attention than the single-class case. To our knowledge, the last significant contribution is that of Stricker and Leonardis  who search for multiple parametric models simultaneously by minimizing description length using Tabu search.
The proposed Multi-X method finds multiple instances of multiple model classes drawing on progress in energy minimization extended with a new move in the label space: replacement of a set of labels with the corresponding density mode in the model parameter domain. Mode seeking significantly reduces the label space, thus speeding up the energy minimization, and it overcomes the problem of multiple instances with similar parameters, a weakness of state-of-the-art single-class approaches. The assignment of data to instances of different model classes is handled by the introduction of class-specific distance functions. Multi-X can also be seen as an extension or generalization of the Hough transform: (i) it finds modes of the parameter space density without creating an accumulator and locating local maxima there, which is prohibitive in high dimensional spaces, (ii) it handles multiple classes – running Hough transform for each model type in parallel or sequentially cannot easily handle competition for data points, and (iii) the ability to model spatial coherence of inliers and to consider higher-order geometric priors is added.
Most recent papers  report results tuned for each test case separately. The results are impressive, but input-specific tuning, i.e. semi-automatic operation with multiple passes, severely restricts possible applications. We propose an adaptive parameter setting strategy within the algorithm, allowing the user to run Multi-X as a black box on a range of problems with no need to set any parameters. Considering that outliers may form structures in the input, as a post-processing step, a cross-validation-based technique removes insignificant instances.
The contributions of the paper are: (i) A general formulation is proposed for multi-class multi-instance model fitting which, to the best of our knowledge, has not been investigated before. (ii) The commonly used energy minimizing technique, introduced by PEARL , is extended with a new move in the label space: replacing a set of labels with the corresponding density mode in the model parameter domain. Benefiting from this move, the minimization is speeded up, terminates with lower energy and the estimated model parameters are more accurate. (iii) The proposed pipeline combines state-of-the-art techniques, such as energy-minimization, median-based mode-seeking, cross-validation, to achieve results superior to the recent multi-model fitting algorithms both in terms of accuracy and processing time. Proposing automatic setting for the key optimization parameters, the method is applicable to various real world problems.
Before presenting the general definition, let us consider a few examples of multi-instance fitting: to find a pair of line instances interpreting a set of 2D points . Line class is the space of lines equipped with a distance function () and a function for estimating from data points. Another simple example is the fitting circle instances to the same data. The circle class is the space of circles, is a distance function and is an estimator. Multi-line fitting is the problem of finding multiple line instances , while the multi-class case is extracting a subset , where . The set is the space of all classes, e.g. line and circle. The formulation includes the outlier class where each instance has constant but possibly different distance to all points , and . Note that considering multiple outlier classes allows interpretation of outliers askk originating from different sources.
The objective of multi-instance multi-class model fitting is to determine a set of instances and labeling assigning each point to an instance minimizing energy . We adopt energy
to measure the quality of the fitting, where and are weights balancing the different terms described bellow, and , and are the data, complexity terms, and the one considering geometric priors, e.g. spatial coherence or perpendicularity, respectively.
is defined in most energy minimization approaches as
penalizing inaccuracies induced by the point-to-instance assignment, where is the distance function of .
Geometric prior term considers spatial coherence of the data points, adopted from , and possibly higher order geometric terms , e.g. perpendicularity of instances. The term favoring spatial coherence, i.e. close points more likely belong to the same instance, is defined as
where are the edges of a predefined neighborhood-graph, the Iverson bracket equals to one if the condition inside holds and zero otherwise, and is a pairwise weighting term. In this paper, equals to one. For problems, where it is required to consider higher-order geometric terms, e.g. to find three perpendicular planes, can be replaced with the energy term proposed in .
A regularization of the number of instances is proposed by Delong et al.  as a label count penalty where is the set of distinct labels of labeling function . To handle multi-class models which might have different costs on the basis of the model class, we thus propose the following definition:
The term controlling the number of instances is
instead of , where is the weight of the weighted multi-class model referred by label .
3Replacing Label Sets
For the optimization of the previously described energy, we build on and extend the PEARL algorithm . PEARL generates a set of initial instances applying a RANSAC-like randomized sampling technique, then alternates two steps until convergence:
(1) Application of -expansion  to obtain labeling minimizing overall energy w.r.t. the current instance set.
(2) Re-estimation of the parameter vector of each model instance in w.r.t. labeling .
In the PEARL formulation, the only way for a label to be removed, i.e. for an instance to be discarded, is to assign it to no data points. Experiments show that (i) this removal process is often unable to delete instances having similar parameters, (ii) and makes the estimation sensitive to the choice of label cost . We thus propose a new move in the label space: replacing a set of labels with the density mode in the model parameter domain.
Multi-model fitting techniques based on energy-minimization usually generate a high number of instances randomly as a first step  (, where is the ground truth instance set). Therefore, the presence of many similar instances is typical. We assume, and experimentally validate, that many points supporting the sought instances in are often assigned in the initialization to a number of instances in with similar parameters. The cluster around the ground truth instances in the model parameter domain can be replaced with the modes of the density (see Figure 1).
Given a mode-seeking function , e.g. Mean-Shift , which obtains the density modes of input instance set in the th iteration. The proposed move is as
where is the labeling in the th iteration and is the optimal labeling which minimizes the energy w.r.t. to instance set . It can be easily seen, that Equation 5 does not break the convergence since it replaces the instances, i.e. the labels, if and only if the energy does not increase. Note that clusters with cardinality one – modes supported by a single instance – can be considered as outliers and removed. This step reduces the label space and speeds up the process.
The proposed approach, called Multi-X, combining PEARL, multi-class models and the proposed label replacement move, is summarized in Alg. ?. Next, each step is described.
1. Instance generation step generates a set of initial instances before the alternating optimization is applied. Reflecting the assumption that the data points are spatially coherent, we use the guided sampling of NAPSAC . This approach first selects a random point, then the remaining ones are chosen from the neighborhood of the selected point. The same neighborhood is used as for the spatial coherence term in the -expansion. Note that this step can easily be replaced by e.g. PROSAC  for problems where the spatial coherence does not hold or favors degenerate estimates, e.g. in fundamental matrix estimation.
2. Mode-Seeking is applied in the model parameter domain. Suppose that a set of instances is given. Since the number of instances in the solution – the modes in the parameter domain – is unknown, a suitable choice for mode-seeking is the Mean-Shift algorithm  or one of its variants. In preliminary experiments, the most robust choice was the Median-Shift  using Weiszfeld-  or Tukey-medians . There was no significant difference, but Tukey-median was slightly faster to compute. In contrast to Mean-Shift, it does not generate new elements in the vector space since it always return an element of the input set. With the Tukey-medians as modes, it is more robust than Mean-Shift . However, we replaced Locality Sensitive Hashing  with Fast Approximated Nearest Neighbors  to achieve higher speed.
Reflecting the fact that a general instance-to-instance distance is needed, we represent instances by point sets, e.g. a line by two points and a homography by four correspondences, and define the instance-to-instance distance as the Hausdorff distance  of the point sets. Even though it yields slightly more parameters than the minimal representation, thus making Median-Shift a bit slower, it is always available as it is used to define spatial neighborhood of points. Another motivation for representing by points is the fact that having a non-homogeneous representation, e.g. a line described by angle and offset, leads to anisotropic distance functions along the axes, thus complicating the distance calculation in the mode-seeking.
There are many point sets defining an instance and a canonical point set representation is needed. For lines, the nearest point to the origin is used and a point on the line at a fixed distance from it. For a homography , the four points are , , , and . The matching step is excluded from the Hausdorff distance, thus speeding up the distance calculation significantly.
The application of Median-Shift never increases the number of instances : The equality is achieved if and only if the distance between every instance pair is greater than the bandwidth. Note that for each distinct model class, Median-Shift has to be applied separately. According to our experience, applying this label replacement move in the first iteration does not make the estimation less accurate but speeds it up significantly even if the energy slightly increases.
3. Labeling assigns points to model instances obtained in the previous step. A suitable choice for such task is -expansion , since it handles an arbitrary number of labels. Given and an initial labeling in the th iteration, labeling is estimated using -expansion minimizing energy . Note that is determined by -expansion in the first step. The number of the model instances is fixed during this step and the energy must decreases: To reduce the sensitivity on the outlier threshold (as it was shown for the single-instance case in ), the distance function of each class is included into a Gaussian-kernel.
4. Model Fitting re-estimates the instance parameters w.r.t. the assigned points. The obtained instance set is re-fitted using the labeling provided by -expansion. The number of the model instances is constant. fitting is an appropriate choice, since combined with the labeling step, it can be considered as truncated norm.
The overall energy can only decrease or stay constant during this step since it consists of three terms: (1) – the sum of the assignment costs minimized, (2) – a function of the labeling , fixed in this step and (3) – which depends on so remains the same. Thus
5. Model Validation considers that a group of outliers may form spatially coherent structures in the data. We propose a post-processing step to remove statistically insignificant models using cross-validation. The algorithm, summarized in Alg. ?, selects times from the inlier points . In each iteration, an instance is estimated from the selected points and its distance to each point is computed. The original instance is considered stable if the mean of the distances is lower than threshold . Note that is the outlier threshold used in the previous sections.
Automatic parameter setting is crucial for Multi-X to be applicable to various real world tasks without requiring the user to set most of the parameters manually. To avoid manual bandwidth selection for mode-seeking, we adopted the automatic procedure proposed in  which sets the bandwidth of the th instance to the distance of the instance and its th neighbor. Thus each instance has its own bandwidth set automatically on the basis of the input.
is set automatically using the approach proposed in  as follows: , where is the size of the minimal sample to estimate the current model, is the point number and is the maximum expected number of instances in the data. Note that this cost is not required to be high since mode-seeking successfully suppresses instances having similar parameters. The objective of introducing a label cost is to remove model instances with weak supports. In practice, this means that the choice of is not restrictive.
Experiments show that the choice of the number of initial instances does not affect the outcome of Multi-X significantly. In our experiments, the number of instances generated was twice the number of the input points.
Spatial coherence weight value performed well in the experiments. The common problem-specific outlier thresholds which led to the most accurate results was: homographies ( pixels), fundamental matrices ( pixels), lines and circles ( pixels), rigid motions (), planes and cylinders ( cm).
First we compare Multi-X with PEARL  combined with the label cost of Delong et al. . Then the performance of Multi-X applied to the following Computer Vision problems is reported: line and circle fitting, 3D plane and cylinder fitting to LIDAR point clouds, multiple homography fitting, two-view and video motion segmentation.
Comparison of PEARL and Multi-X. In a test designed to show the effect of the proposed label move, model validation was not applied and both methods used the same algorithmic components described in the previous section. A synthetic environment consisting of three 2D lines, each sampled at random locations, was created. Then 200 outliers, i.e. random points, were added.
Figure 2 shows the probability of returning an instance number for Multi-X (top-left) and PEARL (bottom-left). The numbers next to the vertical axis are the number of returned instances. The curve on their right shows the probability () of returning them. For instance, the red curve for PEARL on the right of number is close to the probability, while for Multi-X, it is approximately . Therefore, Multi-X more likely returns the desired number of instances. The processing times (top-right), and convergence energies (bottom-right) are also reported. Values are plotted as the function of the initially generated instance number (horizontal axis; ratio w.r.t. to the input point number). The standard deviation of the zero-mean Gaussian-noise added to the point coordinates is pixels. Reflecting the fact that the noise is usually not known in real applications, we set the outlier threshold to pixels. The maximum model number of the label cost was set to the ground truth value, , to demonstrate that suppressing instances exclusively with label cost penalties is not sufficient even with the proper parameters. It can be seen that Multi-X more likely returns the ground truth number of models, both its processing time and convergence energy are superior to that of PEARL.
For Figure 3, the number of the generated instances was set to twice the point number, the threshold was set to pixels. Each reported property is plotted as the function of the noise added to the point coordinates. The same trend can be seen as in Figure 2: Multi-X is less sensitive to the noise than PEARL. It more often returns the desired instances, its processing time and convergence energy are lower.
Simultaneous Line and Circle Fitting is evaluated on 2D edges of banknotes and coins. Edges are detected by Canny edge detector and assigned to circles and lines manually to create a ground truth segmentation.
Each method started with the same number of initial model instances: twice the data point (e.g. edge) number. The evaluated methods are PEARL , T-Linkage 
Multiple Homography Fitting is evaluated on the AdelaideRMF homography dataset  used in most recent publications. AdelaideRMF consists of image pairs of different resolutions with ground truth point correspondences assigned to planes (homographies). To generate initial model instances the technique proposed by Barath et al.  is applied: a single homography is estimated for each correspondence using the point locations together with the related local affine transformations. Table ? reports the results of PEARL , FLOSS , T-Linkage , ARJMC , RCMSA , J-Linkage , and Multi-X. To allow comparison with the state-of-the-art, all methods, including Multi-X, are tuned separately for each test and only the same image pairs are used as in .
Results using a fixed parameter setting are reported in Table ? (results, except that of Multi-X, copied from ). Multi-X achieves the lowest errors. Compared to results in Table ? for parameters hand-tuned for each problem, the errors are significantly higher, but automatic parameter setting is the only possibility in many applications. Moreover, per-image-tuning leads to overfitting.
Two-view Motion Segmentation is evaluated on the AdelaideRMF motion dataset consisting of image pairs of different sizes and the ground truth – correspondences assigned to their motion clusters.
Figure 5 presents example image pairs from the AdelaideRMF motion datasets partitioned by Multi-X. Different motion clusters are denoted by color. Table 1 shows comparison with state-of-the-art methods when all methods are tuned separately for each test case. Results are the average and minimum misclassification errors (in percentage) of ten runs. All results except that of Multi-X are copied from . For Table ?, all methods use fixed parameters. For both test types, Multi-X achieved higher accuracy than the other methods.
Simultaneous Plane and Cylinder Fitting is evaluated on LIDAR point cloud data (see Figure 6). The annotated database
Multi-X is compared with the same methods as in the line and circle fitting section. PEARL and Multi-X fit cylinders and planes simultaneously while T-Linkage and RPA sequentially. Table ? reports that Multi-X is the most accurate in all test cases except one.
Video Motion Segmentation is evaluated on videos of the Hopkins dataset . Motion segmentation in video sequences is the retrieval of sets of points undergoing rigid motions contained in a dynamic scene captured by a moving camera. It can be considered as a subspace segmentation under the assumption of affine cameras. For affine cameras, all feature trajectories associated with a single moving object lie in a 4D linear subspace in , where F is the number of frames .
Table ? shows that the proposed method outperforms the state-of-the-art: SSC , T-Linkage , RPA , Grdy-RansaCov , ILP-RansaCov , and J-Linkage . Results, except for Multi-X, are copied from . Figure 5 shows two frames of the tested videos.
Multi-X is orders of magnitude faster than currently available Matlab implementations of J-Linkage, T-Linkage and RPA. Attacking the fitting problem with a technique similar to PEARL and SA-RCM, it is significantly faster since it benefits from high reduction of the number of instances in the Median-Shift step (see Table ?).
A novel multi-class multi-instance model fitting method has been proposed. It extends an energy minimization approach with a new move in the label space: replacing a set of labels corresponding to model instances by the mode of the density in the model parameter domain. Most of its key parameters are set adaptively making it applicable as a black box on a range of problems. Multi-X outperforms the state-of-the-art in multiple homography, rigid motion, simultaneous plane and cylinder fitting; motion segmentation; and 2D edge interpretation (circle and line fitting). Multi-X runs in time approximately linear in the number of data points, it is an order of magnitude faster than available implementations of commonly used methods. The source code and the datasets for line-circle and plane-cylinder fitting will be made available with the publication.
- Multi-H: Efficient recovery of tangent planes in stereo images.
D. Barath, J. Matas, and L. Hajder. In British Machine Vision Conference, 2016.
- An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.
Y. Boykov and V. Kolmogorov. Pattern Analysis and Machine Intelligence, 2004.
- Fast approximate energy minimization via graph cuts.
Y. Boykov, O. Veksler, and R. Zabih. Pattern Analysis and Machine Intelligence, 2001.
- Robust fitting of multiple structures: The statistical learning approach.
T.-J. Chin, H. Wang, and D. Suter. In International Conference on Computer Vision, 2009.
- Matching with PROSAC-progressive sample consensus.
O. Chum and J. Matas. In Conference onComputer Vision and Pattern Recognition. IEEE, 2005.
- Mean shift: A robust approach toward feature space analysis.
D. Comaniciu and P. Meer. Pattern Analysis and Machine Intelligence, 2002.
- Locality-sensitive hashing scheme based on p-stable distributions.
M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni. In SoCG, 2004.
- Minimizing energies with hierarchical costs.
A. Delong, L. Gorelick, O. Veksler, and Y. Boykov. International Journal of Computer Vision, 2012.
- Fast approximate energy minimization with label costs.
A. Delong, A. Osokin, H. N. Isack, and Y. Boykov. International journal of computer vision, 2012.
- Sparse subspace clustering.
E. Elhamifar and R. Vidal. In Conference on Computer Vision and Pattern Recognition, 2009.
- Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.
M. A. Fischler and R. C. Bolles. Communications of the ACM, 1981.
- Mean shift based clustering in high dimensions: A texture classification example.
B. Georgescu, I. Shimshoni, and P. Meer. In International Conference on Computer Vision, 2003.
- Lower order circle and ellipse hough transform.
N. Guil and E. L. Zapata. Pattern Recognition, 1997.
- Method and means for recognizing complex patterns, 1962.
P. V. C. Hough.
- A survey of the hough transform.
J. Illingworth and J. Kittler. Computer Vision, Graphics, and Image Processing, 1988.
- Energy-based geometric multi-model fitting.
H. Isack and Y. Boykov. International Journal on Computer Vision, 2012.
- Detection of planar regions with uncalibrated stereo using distributions of feature points.
Y. Kanazawa and H. Kawakami. In British Machine Vision Conference, 2004.
- Floss: Facility location for subspace segmentation.
N. Lazic, I. Givoni, B. Frey, and P. Aarabi. In International Conference on Computer Vision, 2009.
- Fixing the locally optimized RANSAC.
K. Lebeda, J. Matas, and O. Chum. In British Machine Vision Conference, 2012.
- Efficient structure detection via random consensus graph.
H. Liu and S. Yan. In Conference on Computer Vision and Pattern Recognition, 2012.
- T-Linkage: A continuous relaxation of J-Linkage for multi-model fitting.
L. Magri and A. Fusiello. In Conference on Computer Vision and Pattern Recognition, 2014.
- Robust multiple model fitting with preference analysis and low-rank approximation.
L. Magri and A. Fusiello. In British Machine Vision Conference, 2015.
- Multiple model fitting as a set coverage problem.
L. Magri and A. Fusiello. In Conference on Computer Vision and Pattern Recognition, 2016.
- Robust detection of lines using the progressive probabilistic hough transform.
J. Matas, C. Galambos, and J. Kittler. Computer Vision and Image Understanding, 2000.
- Fast approximate nearest neighbors with automatic algorithm configuration.
M. Muja and D. G. Lowe. International Conference on Computer Vision Theory and Applications, 2009.
- NAPSAC: High noise, high dimensional robust estimation - it’s in the bag.
D. Nasuto and J. M. B. R. Craddock. 2002.
- Interacting geometric priors for robust multi-model fitting.
T. T. Pham, T.-J. Chin, K. Schindler, and D. Suter. TIP, 2014.
- Simultaneous sampling and multi-structure fitting with adaptive reversible jump mcmc.
T. T. Pham, T.-J. Chin, J. Yu, and D. Suter. In Annual Conference on Neural Information Processing Systems, 2011.
- The random cluster model for robust geometric fitting.
T. T. Pham, T.-J. Chin, J. Yu, and D. Suter. Pattern Analysis and Machine Intelligence, 2014.
- Variational analysis.
R. T. Rockafellar and R. J.-B. Wets. Springer Science & Business Media, 2009.
- Ellipse fitting by accumulating five-point fits.
P. L. Rosin. Pattern Recognition Letters, 1993.
- Mode-detection via median-shift.
L. Shapira, S. Avidan, and A. Shamir. In International Conference on Computer Vision, 2009.
- ExSel++: A general framework to extract parametric models.
M. Stricker and A. Leonardis. In International Conference on Computer Analysis of Images and Patterns, 1995.
- Non-iterative approach for fast and accurate vanishing point detection.
J.-P. Tardif. In International Conference on Computer Vision, 2009.
- Robust multiple structures estimation with j-linkage.
R. Toldo and A. Fusiello. In European Conference on Computer Vision, 2008.
- A benchmark for the comparison of 3-d motion segmentation algorithms.
R. Tron and R. Vidal. In Conference on Computer Vision and Pattern Recognition, 2007.
- Mathematics and the picturing of data.
J. W. Tukey. In ICM, 1975.
- Detecting planar homographies in an image pair.
E. Vincent and R. Laganiére. In International Symposium on Image and Signal Processing and Analysis, 2001.
- Mode-seeking on hypergraphs for robust geometric model fitting.
H. Wang, G. Xiao, Y. Yan, and D. Suter. In International Conference of Computer Vision, 2015.
- Sur le point pour lequel la somme des distances de n points donnés est minimum.
E. Weiszfeld. Tohoku Mathematical Journal, 1937.
- Dynamic and hierarchical multi-structure geometric model fitting.
H. S. Wong, T.-J. Chin, J. Yu, and D. Suter. In International Conference on Computer Vision, 2011.
- A new curve detection method: randomized hough transform (rht).
L. Xu, E. Oja, and P. Kultanen. Pattern Recognition Letters, 1990.
- Nonparametric estimation of multiple structures with outliers.
W. Zhang and J. Kosecká. In Dynamical Vision. 2007.
- The multiransac algorithm and its application to detect planar homographies.
M. Zuliani, C. S. Kenney, and B. Manjunath. In ICIP. IEEE, 2005.