On the Solvability of Viewing Graphs

On the Solvability of Viewing Graphs

Matthew Trager Inria École Normale Supérieure, CNRS, PSL Research University    Brian Osserman UC Davis    Jean Ponce Inria École Normale Supérieure, CNRS, PSL Research University
Abstract

A set of fundamental matrices relating pairs of cameras in some configuration can be represented as edges of a “viewing graph”. Whether or not these fundamental matrices are generically sufficient to recover the global camera configuration depends on the structure of this graph. We study characterizations of “solvable” viewing graphs, and present several new results that can be applied to determine which pairs of views may be used to recover all camera parameters. We also discuss strategies for verifying the solvability of a graph computationally.

Keywords:
Viewing graph, fundamental matrix, 3D reconstruction.

1 Introduction

Multi-view geometry has been studied by photogrammeters since the 1950s [21] and by computer vision researchers since the 1980s [11]. Still, most results to date are concerned with using multi-view tensors to characterize feature correspondences in 2, 3, or 4 views, and determine the corresponding projection matrices [5, 7, 12, 17]. Although correspondences have also been characterized for arbitrary numbers of views [3, 9, 23], very little theoretical work has been devoted to understanding the geometric constraints imposed on configurations of cameras by these tensors, including fundamental matrices [12], which are probably by far the most used in practice. Apart from a few works such as [10, 15], understanding how many and which fundamental matrices can be used to recover globally consistent camera parameters is a largely unexplored problem.

We address this topic in this paper. Following [10], we associate sets of fundamental matrices with edges of a “viewing graph”, and present a series of new results for determining whether a graph is “solvable”, i.e., whether it represents fundamental matrices that determine a unique camera configuration. We also describe effective strategies for verifying solvability, and include some computational experiments using these methods. Our focus here is clearly of a theoretical nature, but understanding how subsets of fundamental matrices constrain the reconstruction process is clearly important in practice as well. Moreover, we will point out that one of our main results (Theorem 3.3) is constructive, and could potentially find applications in reconstruction algorithms (e.g., it could be incorporated in large-scale systems such as [19], that incrementally build up networks of cameras to estimate their parameters).

Previous work. The first investigation of viewing graphs and their solvability can be found in [10]. In that work, Levi and Werman characterize all solvable viewing graphs with at most six vertices, and discuss a few larger solvable examples. Although they provide some useful necessary conditions (see our Proposition 2 and Example 2), they do not address the problem of solvability in general. In [15], Rudi et al. also consider viewing graphs, studying mainly whether a configuration can be recovered from a set of fundamental matrices using a linear system. They also present some “composition rules” for merging solvable graphs into larger ones. Trager et. al [22] provide a sufficient condition for solvability using fundamental matrices, and point to a possible connection with “Laman graphs” and graph rigidity theory. Indeed, Özyesil and Singer [14] show that if one uses essential matrices instead of fundamental ones then solvability can be characterized in terms of so-called “parallel-rigidity” for graphs. Their analysis however does not carry over to the more general setting of uncalibrated cameras. Finally, the viewing graph has also been considered in more practical work: for example, in [18, 20], it is used to enforce triple-wise consistency among fundamental matrices before estimating camera parameters.
Main contributions.
We show that the minimum number of fundamental matrices that can be used to recover a configuration of cameras is always (Theorem 3.1).
We present several criteria for deciding whether or not a viewing graph is solvable. After revisiting some results from [10, 15] (Section 3.1), we describe a new necessary condition for solvability that is based on the number of edges and vertices of subgraphs (Theorem 3.2), as well as a sufficient condition based on “moves” for adding new edges to a graph (Theorem 3.3).
We describe an algebraic formulation for solvability that in principle can always be used to determine whether any viewing graph is solvable. Although this method is computationally challenging for larger graphs, we also introduce a much more practical linear test, that can be used to verify whether a viewing graph identifies a finite number of camera configurations (Section 4).
Using an implementation of all the proposed methods, we analyze solvability for all minimal viewing graphs with at most vertices. We also discuss some relevant examples (Section 5).

2 Background

To make our presentation mostly self-contained, we recall some basic theoretical facts that are used in the rest of the paper.
Notation. We write for the -dimensional real projective space. We use bold font for vectors and matrices, and normal font for projective objects. For example, a point in will be written as where is a vector in and is the equivalence class associated with . Similarly, a projective transformation represented by a matrix will be written as . We use for the group of invertible real matrices.

2.1 Camera configurations and epipolar geometry

A projective camera is represented by matrix of full rank, defined up to scale. The matrix describes a linear projection where is the pinhole of the camera, associated with the null-space of .

The matrix group acts on the set of cameras by multiplication on the right, and represents the group of projective transformations of , or of changes of homogeneous coordinates. We will use the fact that the group of matrices in that fix a camera with pinhole is given by

(1)

Here stands for “stabilizer”. Indeed, all the solutions for in are described by (1). Note that only depends on the pinhole of . The following important fact follows directly from the form of .

Lemma 1

Given two cameras , with distinct pinholes, we have that

(2)

In other words, the identity is the only projective transformation that fixes both and .

Two sets of cameras and with , are projectively equivalent if there exists a single projective transformation such that (so if with in , then for non-zero constants ). The set of configurations of cameras is the set of -tuples of cameras up to projective equivalence. For any , the space of camera configurations can be viewed as a manifold of dimension .

Given two cameras , , the associated fundamental matrix can be defined as the matrix (up to scale) with entries

(3)

where denotes the -th row of , and and are triples of distinct indices. The fundamental matrix can be used to characterize pairs of corresponding points in the two images, since and are projections of the same 3D point if and only if . For our purposes, the most important property of the fundamental matrix is that it is invariant under projective transformations, and that uniquely identifies the configuration of and  [6, Theorem 9.10].

Finally, viewed as a subset of , the (closure of the) set of all fundamental matrices forms a hypersurface defined by . If , the left and right null-space of represent the two epipoles and , which are the images of each pinhole viewed from the other camera. An epipole accounts for two of seven degrees of freedom of a fundamental matrix. In fact, the information encoded in the fundamental matrix can be seen as the pair of epipoles , together with a projective transformation (known as “epipolar line homography” [6]) between lines containing in the first image and the lines containing in the second image. In particular, the knowledge of two epipoles together with three point correspondences completely determines a fundamental matrix.

3 The viewing graph

The viewing graph is a graph in which vertices correspond to cameras, and edges represent fundamental matrices between them. More precisely, if is an undirected graph with vertices, and are projective cameras, we write

(4)

for the set of fundamental matrices defined by the edges of . We say that the the set is solvable if implies that and are in the same projective configuration. In other words, a set of fundamental matrices is solvable if and only if it uniquely determines a projective configuration of cameras.

Proposition 1

The solvability of only depends on the graph and on the pinholes of .

Proof

The statement expresses the fact that changes of image coordinates are only a “relabeling” of a camera configuration and the associated fundamental matrices. More precisely, if are arbitrary projective transformations of , then and are in the same configuration if and only if the same is true for and . This implies that is solvable if and only is. ∎

Example 1

If is a complete graph with vertices, then is solvable if and only if the pinholes of the cameras are not all aligned. Indeed, if the pinholes are aligned, then the fundamental matrices between all pairs of cameras are not sufficient to completely determine the configuration: replacing any with , where is the pinhole of another camera and is arbitrary, yields a new set of cameras which belongs to a different configuration but has the same set of fundamental matrices. Conversely, it is known (see for example [9, 22]) that the complete set of fundamental matrices determines a unique camera configuration whenever there are at least three non-aligned pinholes.

In the rest of the paper we will only consider generic configurations of cameras/pinholes (so a complete graph will always be solvable). This covers most cases of practical interest, although in the future degenerate configurations (including some collinear or coplanar pinholes) could be studied as well.

Definition 1

A viewing graph is said to be solvable if is solvable for generic cameras .

In other words, solvable viewing graphs describe sets of fundamental matrices that are generically sufficient to recover a camera configuration. Despite its clear significance, the problem of characterizing which viewing graphs are solvable has not been studied much, and only partial answers are available in the literature (mainly in [10, 15]). It is quite easy to produce examples of graphs that are solvable, but it is much more challenging, given a graph, to determine whether it is solvable or not. The following observation provides another useful formulation of solvability (note that the “if” part requires the genericity assumption, as shown in Example 1).

Lemma 2

A viewing graph is solvable if and only if, for generic cameras , the fundamental matrices uniquely determine the remaining fundamental matrices .

This viewpoint also suggests the idea that, given any graph , we can define a “solvable closure” , as the graph obtained from by adding edges corresponding to fundamental matrices that can be deduced generically from . Hence, a graph is solvable if and only if its closure is a complete graph. We will return to this point in Section 3.4.

3.1 Simple criteria

We begin by recalling two necessary conditions for solvability that were shown in [10]. These provide simple criteria to show that a viewing graph is not solvable.

Proposition 2

[10] If a viewing graph with vertices is solvable, then: 1) All vertices have degree at least 2. 2) No two adjacent vertices have degree 2.

We extend this result with the following necessary condition (which implies the first point in the previous statement).

Proposition 3

Any solvable graph is 2-connected, i.e., it has the property that after removing any vertex the graph remains connected.

Proof

Assume that a vertex disconnects the graph into two components , and let be a set of generic cameras, whose pairwise fundamental matrices are represented by the edges of . If is the pinhole of the camera associated with , then we consider two distinct projective transformations of the form and . These transformations fix the camera . If we apply to all cameras in and to all cameras in , while leaving fixed, we obtain a different camera configuration that gives rise to the same set of fundamental matrices as for all edges in . ∎

We also recall a result from [15] which will be used in the next section.

Proposition 4

[15] If and are solvable viewing graphs, then the graph obtained by identifying two vertices from and with two from is solvable.

Note that if both pairs of vertices in the previous statement are connected by edges in and , then these two edges will automatically be identified in .

3.2 How many fundamental matrices?

We now ask ourselves what is the minimal number of edges that a graph must have to be solvable (or, equivalently, how many fundamental matrices are required to recover a camera configuration). Since a single epipolar relation provides at most constraints in the -dimensional space camera configuration, we deduce that any solvable graph must have at least edges. This fact was previously observed in [15, Theorem 2]. However, compared to [15], we show here that this bound is tight, i.e., that there always exists a solvable graph with edges. Concretely, this means that, for generic views, there is always a way of recovering the corresponding camera configuration using fundamental matrices.

Theorem 3.1

The minimum number of edges of a solvable viewing graph with views is

Proof

For , examples of solvable viewing graphs with edges are illustrated Figure 1. The solvability of these graphs will be shown in Section 3.4 (all but one of these also appear in [10]). In particular, let be a solvable viewing graph with vertices and edges. Using Proposition 4, we deduce that, starting from a solvable viewing graph with vertices and edges, we can always construct a solvable graph with vertices and edges. The graph is simply obtained by merging and as in Proposition 4, using two pairs of vertices both connected by edges.

Now, for any , we consider the unique integers such that and . It is easy to see that

To obtain a solvable viewing graph with vertices and edges, we start from a solvable graph with vertices and edges, and repeat the gluing construction described above times. The resulting graph is solvable and has the desired number of vertices and edges. ∎

Remark 1

It is worth pointing out that, in order to recover projection matrices for views, it is quite common to use fundamental matrices (see for example [8, Sec.4.4]). In fact, as shown in [22, Proposition 7], a large class of solvable viewing graphs can be defined, starting for example from a -cycle, by adding vertices of degree two, one at the time: this always gives a total of edges. For this type of viewing graphs it is possible to recover projection matrices incrementally, using a pair of fundamental matrices for each camera. In fact, it is probably quite often erroneously believed that is the minimal number of fundamental matrices that are required for multi-view reconstruction. Part of the confusion may arise from the fact that the “joint image” [23, 22, 1], which characterizes multi-view point correspondences in , has dimension three (or codimension ). This means means that we expect conditions to be necessary to cut out generically the set of image correspondences among views. On the other hand, according to Theorem 3.1, fewer constraints are actually sufficient to determine camera geometry.111This implies however that fewer than conditions can in fact determine a joint image in , at least “indirectly” through the camera configuration. Mathematically, this is an interesting phenomenon that could be investigated in the future.

Some values of are listed in Table 1 (here represents the minimal number of constraints on the fundamental matrices, and will be discussed in the next section). Note that for all .

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 3 5 6 8 9 11 12 14 16 17 19 20 22 23
0 3 6 2 5 1 4 0 3 6 2 5 1 4 0
Table 1: The relation between , , and

Figure 1: Examples of minimal solvable viewing graphs for views (see Section 3.4)

3.3 Constraints on fundamental matrices

Closely related to the solvability of viewing graphs is the problem of describing compatibility of fundamental matrices. Indeed, given a solvable graph , it is not true in general that any set of fundamental matrices can be assigned to the edges of , since fundamental matrices must satisfy some feasibility constraints in order to correspond to an actual camera configuration. For example, it is well known that the fundamental matrices relating three pairs of cameras with non-aligned pinholes are compatible if and only if

(5)

where is the epipole in image relative to the camera  [6, Theorem 15.6]. In most practical situations fundamental matrices are estimated separately, so these constraints need to be taken into account [20]. However, it is sometimes incorrectly stated that compatibility for any set of fundamental matrices only arises from triples and equations of the form (5[15, Theorem 1], [20, Definition 1]. While it is true that for a complete set of fundamental matrices triple-wise compatibility is sufficient to guarantee global compatibility, for smaller sets of fundamental matrices other types of constraints will be necessary. For example, there are many solvable viewing graphs with no three-cycles (e.g., the graph in Figure 1 with ), however the fundamental matrices cannot be unconstrained if , which always true unless modulo (cf. Table 1).

More formally, we can consider the set of compatible fundamental matrices between all pairs of views, so that where . Since each compatible -tuple is associated with a camera configuration, we see that has dimension . Given a viewing graph with views, we write for the projection of onto the factors in corresponding to the edges of . The set thus represents compatible fundamental matrices for pairs of views associated with the edges of . The following result follows from dimensionality arguments (see the supplementary material for a complete proof).

Proposition 5

If is solvable with vertices, has dimension .

If has dimension , then the fundamental matrices assigned to the edges of must satisfy constraints222This is the codimension of in where is the determinant hypersurface.. This also means that the minimum number of constraints on the fundamental matrices associated with a solvable graph is (see Table 1).

We now use Proposition 5 to deduce a new necessary condition for solvability.

Theorem 3.2

Let be a solvable graph with vertices and edges. Then for any subgraph of with vertices and edges we must have

(6)

where . More generally, if are subgraphs of , each with vertices and edges, with the property that the edge sets are pairwise disjoint, then we must have

(7)
Proof

Using the same notation as above, we note that that is a projection of onto factors of : this implies , or . Since and (because is solvable), we obtain

For the second statement, we consider the graph . Since the edges of are disjoint, we have

and . The result follows again from . ∎

Example 2

In [10], Levi and Werman observe that all viewing graphs of the form shown in Figure 2 are not solvable. This can be easily deduced from Theorem 3.2. Indeed, for a graph of this form, the subgraphs have disjoint edges, however we have (using the same notation as in the proof of Theorem 3.2)

According to Theorem 3.2 this means that is not solvable.

Figure 2: A viewing graph of this form (where represent arbitrary subgraphs) is not solvable

3.4 Constructive approach for verifying solvability

Until now we have mainly discussed necessary conditions for solvability, which can be used to show that a given graph is not solvable. We next introduce a general strategy for proving that a graph is solvable. This method is not always guaranteed to work, but in practice it gives sufficient conditions for most of the graphs we tested (cf. Section 5).

Recall from the beginning of this section that we introduced the “viewing closure” of as the graph obtained by adding to all edges corresponding to fundamental matrices that can be deduced from . Our approach consists of a series of “moves” which describe valid ways to add new edges to a viewing graph. For this it is convenient to introduce a new type of edge in the graph, which keeps track of the fact that partial information about a fundamental matrix is available. More precisely:

  • A solid (undirected) edge between vertices and means that the fundamental matrix between the views and is fixed (as before).

  • A directed dashed edge (for short, a dashed arrow) between vertices and means that the -th epipole in the image is fixed.

As these definitions suggest, a solid edge also counts as a dashed double- arrow, but the converse is not true. We next introduce three basic “moves” (cf. Figure 3).

  1. If there are solid edges defining a four-cycle with one diagonal, draw the other diagonal.

  2. If there are dashed arrows , , and solid edges and , draw a dashed arrow .

  3. If there are double dashed arrows , together with three pairs of dashed arrows for , make the arrow between and a solid (undirected) edge.

Figure 3: Three moves (left: I, center: II, right: III) that can be used to prove solvability
Theorem 3.3

Let be a viewing graph. If applying the three moves described above iteratively to we obtain a complete graph, then is solvable.

Proof

For each of the three moves we need to show that the new edges contain information about the unknown fundamental matrices that can actually be deduced from .

Move I: The second diagonal of the square is deducible from the other edges because the square with one diagonal is a solvable graph (this is a simple consequence of Proposition 4).

Move II: Assume that and are fixed epipoles in images and , and that the fundamental matrices are also fixed. If are not aligned, we can use to “transfer” the point , and obtain a line in image that contains epipole . Similarly, if are not aligned, we obtain another line using the same procedure with and . If the pinholes are not all coplanar, the lines and will be distinct, and their point of intersection will be . This implies that we can draw a dashed arrow from to .

Move III: Assume that the epipoles and are fixed, and that the images of three other pinholes are fixed in both images and . If the planes for are distinct, then the images of give three correspondences that fix the epipolar line homography. This completely determines , and we can draw a solid edge between and . ∎

In practice, the three moves can be applied cyclically until no new edges can be added (it is also easy to argue the order is irrelevant, because we are simply annotating information that is always deducible from the graph). Finally, we note that all three moves are constructive and linear, meaning they actually provide a practical strategy for computing all fundamental matrices: it is enough to transfer epipoles appropriately, and use them to impose linear conditions on the unknown fundamental matrices.

Example 3

Using Theorem 3.3, we can show that all graphs from Figure 1 are solvable. Figure 4 illustrates this explicitly for two cases ( and ).

II (2x)

III (2x)

I (6x)


II (6x)

III (3x)

I (9x)

Figure 4: Two applications of Theorem 3.3 to prove that viewing graphs are solvable

4 Algebraic tests for solvability and finite solvability

Given a viewing graph , it is possible to write down a set of algebraic conditions that will in principle always determine whether is solvable. One way to do this is by characterizing the set of projective transformations of that can be applied to all cameras without affecting any of the fundamental matrices represented by the edges of the viewing graph. More precisely, since every pair of vertices connected by an edge represents a projectively rigid pair of cameras, we assign a matrix in to each edge of the graph (so describes a projective transformation applied to a pair of cameras). We then impose that matrices on adjacent edges act compatibly on the shared vertex/camera. If the edges and share a vertex , then from (1) we see that this compatibility can be written as

(8)

where is an arbitrary (nonzero) constant and is an arbitrary vector. Thus, if is a viewing graph with edges and are a set of pinholes, we consider the set of all compatible assignments of matrices:

If is solvable, then for general the set will consist of -tuples of matrices that are all scalar multiples of each other. This in fact means that the only way to act on all cameras without affecting the fixed fundamental matrices is to apply a single projective transformation.

By substituting random pinholes in (8), we can use these equations for as an algebraic test for verifying whether a viewing graph is solvable. This approach however is computationally very challenging, since it requires solving a non-linear algebraic system with a large number of variables. On the other hand, if we are only interested in the dimension of , then we can use a much simpler strategy: noting that may be viewed as an algebraic group (it is a subgroup of ), it is sufficient to compute the dimension of its tangent space at any point, and in particular at the identity (i.e., the product of identity matrices).333Here we actually need that is smooth: this follows from a technical result, which states that an algebraic group (more properly a “group scheme”) over a field of characteristic zero is always smooth [13, Sec.11]. An explicit representation of the tangent space of is provided by the following result (see the supplementary material for a proof).

Proposition 6

The tangent space of at the identity can be represented as the space of -tuple of matrices where each is in (not necessary invertible), and with compatibility conditions of the form

(9)

where and are arbitrary, and and share the vertex .

When the pinholes have been fixed, the compatibility constraints (9) can be expressed as linear equations in the entries of the matrices . These equations are obtained by eliminating the variables and from (9). The resulting conditions in terms of are rather simple, and listed explicitly in the supplementary material. Using this approach, the dimension of is easy to determine: it is enough to fix the pinholes randomly, and compute the dimension of the induced linear system.

When has dimension (which accounts for the group of projective transformations, and scale factors for each matrix ), we deduce that there are at most a finite number of projectively inequivalent ways in which we can act on all the cameras without affecting the fixed fundamental matrices. In other words, the fundamental matrices associated with the edges of determine at most a finite set of camera configurations (rather than a single configuration, which is our definition for solvability). When this happens, we say that is finite solvable. On the other hand, we were not able to find an example of a finite solvable graph that is provably not solvable, nor to find a proof that “finite solvability” implies “solvability”. To our knowledge, whether a set of fundamental matrices can characterize a finite number of configurations, but more than a single one, is a question that has never been addressed.

Open Question

Is it possible for a viewing graph to be finite solvable without being solvable?

Our experiments show that this behavior does not occur for a small number of vertices, but we see no reason why this should be true for larger graphs. This is certainly an important issue that we hope to investigate in the future.

5 Experiments and examples

We have implemented and tested all of the discussed criteria and methods using the free mathematical software SageMath [2].444Our code is available at https://github.com/mtrager/viewing-graphs. We then analyzed solvability for all minimal viewing graphs with vertices and edges. The results are summarized in Table 2. For every pair , we list the number of all non-isomorphic connected graphs of that size (“connected”), the number of graphs that satisfy the necessary condition from Theorem 3.2 (“candidates”), the number of those that satisfy the sufficient condition from Theorem 3.3 (“solvable with moves”), and the number of graphs that are finite solvable (“finite solvable”), using the linear method from 4. We see that Theorems 3.2 and 3.3 allow us to recover all minimal solvable graphs for , since candidate graphs are always solvable with moves. On the other hand, for , and particularly for the unconstrained case , there are some graphs that we could not classify with those methods (although finite solvability was easy to verify in all cases). For the undecided graphs, we were sometimes able to prove solvability with the general algebraic method from Section 4, or using other arguments. The following examples present a few interesting cases.

(3,3) (4,5) (5,6) (6,8) (7,9) (8,11) (9,12)
connected 1 1 5 22 107 814 4495
candidates 1 1 1 4 3 36 28
solvable with moves 1 1 1 4 3 31 5
finite solvable 1 1 1 4 3 36 27
Table 2: Solvability of minimal viewing graphs using our methods
Example 4

The graph shown in Figure 5 (left) is one of the five cases with , that are “candidates” but are not “solvable with moves”. However, we can show that this graph is actually solvable by arguing that the image of the pinhole in the view is fixed, even if this is not a consequence of the moves of Theorem 3.3 (this is represented by the gray dashed arrow in the figure). To prove this fact, one needs to keep track of more information, and record also when an epipole is constrained to a line (rather than only when an epipole is fixed, which is the purpose of dashed edges).555This information can be taken into account by defining a new type of edge together with additional moves. We did not do this in Theorem 3.3 because this type of edge is never necessary for smaller graphs. After drawing the dashed arrow from to , solvability can be shown using the moves from Theorem 3.3.

Example 5

The graph shown in Figure 5 (center) is the only viewing graph with and that is “candidate” but is not “finite solvable”. The fact that it is not finite solvable can also deduced without computations. Indeed, any finite solvable viewing graph of this size cannot impose any constraints on the fundamental matrices associated with its edges (this is because ). However, the image of the pinhole in the view is over-constrained, because we can draw a dashed arrow using move II for two distinct four-cycles (() and ()). This implies that the fundamental matrices associated with the edges of the graph cannot be arbitrary. 

Example 6

The graph shown in Figure 5 (right) is not “solvable with moves”, however one can show that it is solvable: indeed, the general algebraic compatibility equations from Section 4 are in this case simple and can be solved explicitly (see the supplementary material for the computations). The fundamental matrices associated with the edges of the graph are unconstrained, so arbitrary fundamental matrices determine a unique configuration of  cameras. 

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

9

Figure 5: The graphs described in Examples 4, 5, 6

6 Conclusions

We have studied the problem of solvability of viewing graphs, presenting a series of new theoretical results that can be applied to determine whether a graph is solvable. We have also pointed out some open questions (particularly, the relation between finite solvability and solvability, discussed in Section 4), and we hope that this paper can lead to further work on these issues.

Our main focus here was to understand whether the camera-estimation problem is well-posed, and we did not directly address the task of determining the configuration computationally. Properly recovering a global camera configuration that is consistent with local measurements is challenging, and is arguably the main obstacle for any structure-from-motion algorithm. For this reason, we believe that a complete understanding of the algebraic constraints that characterize the compatibility of fundamental matrices would be very useful. This is an issue that has not been considered much in classical multi-view geometry, and is very closely related to the topic of this paper. We plan to investigate it next.

Acknowledgments.

This work was supported in part by the ERC grant VideoWorld, the Institut Universitaire de France, the Inria-CMU associated team GAYA, a collaboration agreement between Inria and NYU, and a grant from the Simons Foundation #279151.

References

  • [1] Aholt, C., Sturmfels, B., Thomas, R.: A hilbert scheme in computer vision. Canad. J. Math 65(5), 961–988 (2013)
  • [2] Developers, T.S.: SageMath, the Sage Mathematics Software System (Version 8.0.0) (2017), http://www.sagemath.org
  • [3] Faugeras, O., Mourrain, B.: On the geometry and algebra of the point and line correspondences between n images. In: Computer Vision, 1995. Proceedings., Fifth International Conference on. pp. 951–956. IEEE (1995)
  • [4] Grothendieck, A.: Éléments de géométrie algébrique iv. Publ. Math. IHES, Ibid. 24 (1965)
  • [5] Hartley, R.: Computation of the quadrifocal tensor. pp. 20–35 (1998)
  • [6] Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
  • [7] Hartley, R.I.: Lines and points in three views and the trifocal tensor. International Journal of Computer Vision 22(2), 125–140 (1997)
  • [8] Heyden, A.: Tensorial properties of multiple view constraints. Mathematical Methods in the Applied Sciences 23(2), 169–202 (2000)
  • [9] Heyden, A., Åström, K.: Algebraic properties of multilinear constraints. Mathematical Methods in the Applied Sciences 20(13), 1135–1162 (1997)
  • [10] Levi, N., Werman, M.: The viewing graph. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on. vol. 1, pp. I–I. IEEE (2003)
  • [11] Longuet-Higgins, H.C.: A computer algorithm for reconstructing a scene from two projections. Nature 293(5828),  133 (1981)
  • [12] Luong, Q.T., Faugeras, O.: The fundamental matrix: theory, algorithms, and stability analysis 17(1), 43–76 (1996)
  • [13] Mumford, D.: Abelian Varieties. Studies in mathematics, Hindustan Book Agency (2008), https://books.google.it/books?id=J81jPwAACAAJ
  • [14] Ozyesil, O., Singer, A.: Robust camera location estimation by convex programming. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2674–2683 (2015)
  • [15] Rudi, A., Pizzoli, M., Pirri, F.: Linear solvability in the viewing graph. In: Asian Conference on Computer Vision. pp. 369–381. Springer (2010)
  • [16] Shafarevich, I.R., Hirsch, K.A.: Basic algebraic geometry, vol. 1. Springer (1994)
  • [17] Shashua, A.: Algebraic functions for recognition 17(8), 779–789 (1995)
  • [18] Sinha, S.N., Pollefeys, M.: Camera network calibration and synchronization from silhouettes in archived video. International journal of computer vision 87(3), 266–283 (2010)
  • [19] Snavely, N., Seitz, S., Szeliski, R.: Photo tourism: Exploring image collections in 3D. In: SIGGRAPH (2006)
  • [20] Sweeney, C., Sattler, T., Hollerer, T., Turk, M., Pollefeys, M.: Optimizing the viewing graph for structure-from-motion. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 801–809 (2015)
  • [21] Thompson, M., Eller, R., Radlinski, W., Speert, J. (eds.): Manual of Photogrammetry. American Society of Photogrammetry (1966), third Edition.
  • [22] Trager, M., Hebert, M., Ponce, J.: The joint image handbook. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 909–917 (2015)
  • [23] Triggs, B.: Matching constraints and the joint image. In: Computer Vision, 1995. Proceedings., Fifth International Conference on. pp. 338–343. IEEE (1995)

This supplementary material contains some technical discussions and examples that were not included in the main body of the paper.

Appendix A Solvability using algebraic geometry

It is useful to revisit solvability from a slightly more technical viewpoint. We let be the rational map associating to a set of cameras the fundamental matrices corresponding to the edges of (so that ).666We recall that rational map between projective spaces is a map whose coordinates are homogeneous polynomial functions. The map is in general well-defined only on an open set of the domain. Note that the definition of actually requires fixing an orientation for each edge in , since a fundamental matrix represents an ordered pair of cameras. However, it is easy to see that the orientations of the edges can be chosen arbitrarily. The map is rational because the coefficients of the fundamental matrix are polynomials in the entries of the two projection matrices. The (closure of the) image of this map is an algebraic variety in representing the set of “compatible” fundamental matrices that can be assigned to the edges of . We denote this set with (as in the main part of the paper).

The solvability of the graph can be understood in terms of and . We will use the following general property of rational maps.

Proposition .7

Let be rational map of projective varieties over an algebraically closed field, and let be the closure of the image of . Then there exists a (Zariski) open dense neighborhood of such that the fiber of over has always: 1) the same (pure) dimension , and 2) the same number of connected components.

Proof

The first point is [16, Theorem 7], the second one is [4, Proposition 9.7.8]. ∎

This result states that, over an algebraically closed field, pre-images of general points will have the same dimension and the same number of connected components. In our setting, this implies that if we view as a map on , there exists an open dense set in its image that defines its general behavior, and one of the following holds:

  1. the pre-image of every element in is a unique camera configuration,

  2. the pre-image of every element in contains a fixed finite number of (complex) camera configurations,

  3. the pre-image of every element in contains infinitely many (complex) camera configurations.

Because uniqueness over is stronger than uniqueness over , we see that necessarily if 1 occurs, the graph is solvable, and similarly if either 1 or 2 occur, is finite solvable. On the other hand, it might be possible for to be solvable in case , but this would require that exactly one of the complex configurations in the general fiber is real.

The next result essentially shows that is finite solvable if and only if we are in situations 1 and 2 (because if a set of fundamental is compatible with an infinite set of complex configurations, it is also compatible with an infinite set of real ones).

Proposition .8

Suppose that we are in situation 3 above. Then there is a possibly smaller open dense subset such that every real fiber either contains no real configurations or infinitely many real configurations. Moreover, the latter case occurs over a Zariski dense subset of .

Proof

According to the 2nd Bertini Theorem [16, Theorem 2], there is an open subset of the image of over which the fibers are smooth. If these fibers have any real points, then the implicit function theorem from multivariable calculus implies that real points occur in dimension equal to the complex dimension. ∎

As a corollary we obtain the following result (which implies Proposition 5 in the main part of the paper).

Proposition .9

The graph is finite solvable if and only if has dimension .

Proof

It is sufficient to note that if 1 or 2 occur, the generic fiber of has dimension in . In situation 3, the fiber has higher dimension. ∎

Appendix B Linear equations for finite solvability

We recall here that was defined in the main part of the paper as the set of -tuples of matrices in which satisfied compatibility equations of the form

(10)

where is an arbitrary (nonzero) constant and is an arbitrary vector. The following result will be useful.

Proposition .10

If is a non-zero vector, then a matrix can be written in the form for some arbitrary and if and only if the following linear expressions vanish:

(11)
Proof

The result is easily shown using a computer algebra system. Inside the ring , we consider the ideal obtained by eliminating the variables and from the coordinates of . We can then verify that (11) generate an ideal that decomposes into two prime components: one of these is irrelevant for us (it describes the vanishing of ) and the other one is . ∎

We can now prove Proposition 6 from the main part of the paper.

Proposition 6.The tangent space of at the identity can be represented as the space of -tuples of matrices where each is in (not necessary invertible), and with compatibility conditions of the form

(12)

where and are arbitrary, and and share the vertex .

Proof

According to Proposition .10, a matrix can be written in the form for some and if and only if it satisfies a set linear equations that depend on . Let us write for these linear conditions (so is a linear map). Note that necessarily .

A constraint of the form (10) can now be expressed as . Writing the first order expansion or at we obtain ( denotes equality up to higher order terms)

(13)

This shows that is the tangent space at the identity of each constraint (10). By Proposition .10 we have that is equivalent to (12), and this concludes the proof. ∎

Finally, we note that by substituting inside (11), we obtain explicit equations that can be used to determine finite solvability (as explained in Section 4 of the paper).

Appendix C Examples

Example .7

Let be the four-cycle shown in Figure 6. We immediately see that the graph is not solvable, because a solvable graph with four vertices must have at least edges. It is however useful to understand this example algebraically. Following the general approach described in Section 4 of the paper, we assign the identity to the edge , and unknown matrices , , to the remaining edges. The compatibility equations yield

(14)

which imply that

(15)

Expanding (15) we obtain

(16)

where

(17)

1

4

3

2

Figure 6: The four-cycle from Example .7

From (16) we see that the vectors must be scalar multiples of the rows of the matrix where