Complete Endomorphisms in Computer Vision

# Complete Endomorphisms in Computer Vision

Applied to multiple view geometry

## Abstract

Correspondences between -tuples of points are key in multiple view geometry and motion analysis. Regular transformations are posed by homographies between two projective planes that serves as structural models for images. Such transformations can not include degenerate situations. Fundamental or essential matrices expand homographies with structural information by using degenerate bilinear maps. The projectivization of the endomorphisms of a three-dimensional vector space includes all of them. Hence, they are able to explain a wider range of eventually degenerate transformations between arbitrary pairs of views. To include these degenerate situations, this paper introduces a completion of bilinear maps between spaces given by an equivariant compactification of regular transformations. This completion is extensible to the varieties of fundamental and essential matrices, where most methods based on regular transformations fail. The construction of complete endomorphisms manages degenerate projection maps using a simultaneous action on source and target spaces. In such way, this mathematical construction provides a robust framework to relate corresponding views in multiple view geometry.

###### Keywords:
Epipolar Geometry Essential Matrix Fundamental Matrix Degeneracies Secant Varieties Adjoint Representation

## 1 Introduction

A classical issue regarding 3D reconstruction and motion analysis concerns the preservation of the continuity of the scene or the flow, although small changes in input occurs. There are a lot of answers where different constraints have been introduced from the early eighties. The most common constraint is the structural bilinear - or multilinear - tensor created with -tuples of corresponding elements (points and/or lines) for the camera pose. All these constraints are very sensitive to noise. Classical approaches are based on minimal solutions extracted from noise measurements following a Random Sample Consensus (RANSAC) scheme. However, indeterminacies persist for some degenerate situations - created by low rank matrices - that can arise for mobile cameras.

Classic literature in computer vision faugeras1993three; hartley2003multiple display a low attention to degenerate cases in structural models. These cases arise when independence conditions for features are not fulfilled, including situations where the camera turns around its optical axis or it is in fronto-parallel position w.r.t. a planar surface (a wall or the ground, e.g.). Then, the problem is ill-posed, and conventional solutions consists in performing a “small perturbation” or reboot the process. Both strategies display issues concerning the lack of control about the perturbation to be made that generate undesirable discontinuities. Thus, it is important to develop alternative strategies which can maintain some kind of “coherence” by reusing the “recent history” of the trajectory. History is continuously modeled in terms of generically regular conditions for tensors in previously sampled images with a discrete approach of a well-defined path in the space of structural tensors. Unfortunately, degeneracy conditions for typical features give indeterminacy for limits of structural tensors, which must be removed. Our approach consists of considering Kinematic information of the matrix version of the gradient field for indeterminacy loci.

Less attention has been paid to preserve the “continuity” of eventually singular trajectories in the space of bilinear maps linked to the automatic correspondence between pairs of views. In this case, singular maps are the responsible for indeterminacies in tensors and lie on singular strata of the space of bilinear mapsthorup1988complete. In this work, we develop a more down-to-earth approach using some basic properties of the projectivization of spaces of endomorphisms, including homographies , fundamental and essential matrices. All of them can be described in terms of orbits by a group action on the space of endomorphisms , i.e. linear maps of a vector space in itself. Their simultaneous algebraic treatment allows to extend the algebraic completion to the space of eventually degenerated central projection maps with center . Intuitively, the key for the control of degenerate cases is to select appropriate limits of tangents in a “more complete” space.

Therefore, the main goal of our work is to model a continuous solution that also considers degenerated cases for simplest tensors (fundamental and essential matrices, e.g.) such those appearing in structural models for 3D Reconstruction. In order to achieve this goal, we introduced an “equivariant compactification” of the space of matrices w.r.t. group actions linked to kinematic properties visualized in the dual space. This double representation (positions and “speed”) stores the “recent history” represented by a path in the tangent bundle to for the trajectory of a mobile camera . Our approach is based on a dual presentation for the rank stratification of matrices. This dual representation encodes tangential information at each point represented by the adjoint matrix, a -matrix whose entries represent the gradient of the determinant of .

In the simplest case, after fixing a basis of , endomorphisms of a 3D vector space are given by arbitrary -matrices; they are naturally stratified by the rank giving three orbits with rank . In particular, from the differential viewpoint, sets of homographies and regular (i.e. rank 2) fundamental matrices can be considered as two -orbits of the Lie algebra of by the action of the projectivization of the general linear group corresponding to rank and rank matrices, respectively. More generally, the description of as a union of orbits by the action of gives a structure as an “orbifold”, i.e. a union of -orbits containing their degenerate cases, which are usually excluded from the analysis. They are recovered by introducing a “compactification” where degenerate cases are managed in terms of successive envelopes by linear subspaces of . All arguments can be extended to higher dimensions and even to hypermatrices representing more sophisticated tensors. However, for simplification purposes, we constraint ourselves only to endomorphisms extending planar or spatial homographies to the singular case.

The rest of the paper is organized as follows. Section 2 provides the mathematical background to understand the rest of the paper and frames our approach in the state of the art. Section 3 analyzes the simplest cases including regular transformations defined by homographies for the planar case that relate 2D views using the fundamental variety. Section 4 extends the approach to rigid transformations in the third dimension, including metric aspects in terms of the essential variety involving source and target spaces in . Section 5 studies the structural connection between them using the simultaneous action on source and target spaces of a variable projection linked to the camera pose; left-right and contact -equivalences are explained. Section 6 provides additional insight concerning the details and practical considerations for implementing this approach in Visual Odometry (VO) systems. Finally, Section 7 concludes the paper with a summary of the main results and guidelines for further research.

## 2 Background

Local symmetries are ubiquitous in a lot of problems in Physics and Engineering involving propagation phenomena. Most approaches in applied areas consider only regular regions, by ignoring any kind of degenerations linked to rank deficient matrices linked to linearization of phenomena. To include them, we develop a “locally symmetric completion” of eventually singular transformations for involved tensors such those appearing in Reconstruction issues. Our approach is not quite original; a similar idea can be found in tron2017space, which introduces a locally symmetric structure in a differential framework concerning geodesics on the essential manifold. Nevertheless, the initial geometric description as symmetric space (union of orbits linked to the rank preservation) can not be extended to include a differential approach to degenerate cases. Due to the occurrence of singularities, the support given as a quotient variety is not a smooth manifold but a singular algebraic variety where the methods for Riemannian manifolds no longer apply.

A larger description of a locally symmetric structure may be performed by extending the ordinary algebraic approach. Roughly speaking, it suffices to add limits of tangent subspaces along different “branches” at singularities and “extend the action” to obtain a more complete description including kinematic aspects. In this way, one obtains a “local replication” compatible with the presence of singularities in the adherence of “augmented” orbits by tangent spaces at regular points. So, for the “subregular” case - codimension one orbit - it suffices to construct pairs of eventually degenerate transformations involving the original one and a “generalized dual” transformation (given by the adjoint map in the regular case) representing neighbor tangent directions. So, first order differential approaches of eventually degenerate maps allow to propagate - and consequently, anticipate - partial representations of expected views, even in presence of rank-deficient matrices.

This extended duality allows a simultaneous treatment of incidence and tangency conditions (both are projectively invariant), and to manage degenerate cases in terms of “complete objects” as limits in an enlarged space (including the original space and their duals) which can be managed as a locally symmetric space in terms of extended transformations (original ones and their exterior powers). Besides its differential description as a gradient in the space of matrices, a more geometric description can be developed in terms of pairs of loci and their envelopes. The extended transformations act on the source or ambient space (right action), and on its dual space which can be considered as a target space representing envelopes by tangent subspaces. This idea is reminiscent of the contact action which preserves the graph and it provides a natural extension of the right-left action (see next paragraph). A discrete version of last action has been used by Kanade, Tomasi and Lucas Kanade-Lucas-Tomasi (KLT) along the early nineties in regard to Structure from Motion approaches to 3D Reconstruction. Both actions are commonly used for the infinitesimal classification of map-germs in differential classification of map-germs. However, our approach is more focused towards a local description of the space of generalized transformations and/or projection maps as a locally symmetric space. This structure has the additional advantage of allowing the extension of Riemannian properties given in terms of geodesics.

The simplest simultaneous action on source and target spaces is the Cartesian or direct product of actions. It is denoted by the -action where is the right-left action. Its orbits are given by the double conjugacy classes from the algebraic viewpoint. The -action is very useful for decoupled models (implicit in KLT algorithms or Structure from Motion (SfM), e.g.), and consequently very useful by computational reasons. Despite the wide interest for the above approaches, the -action is less plausible than the -action, which incorporates the graph preservation (corresponding to quadratic contact between a manifold and its tangent space at each contact point ) as the structural constraint.

In our case, contact equivalence is based on a coupling between images and scene representations. Although contact equivalence is well known in Local Differential Topology, its use in Computer Vision is very scarce. It is implicitly embedded in some recognition approaches where one exchanges information about control points and envelopes. However, to our best knowledge, it has not been applied to multiple view geometry issues. We constraint ourselves to almost generic phenomena given by low-corank singularities. In this way, a more stable “geometric control” of limit positions using envelopes of linear subspaces can be performed.

## 3 Completing planar homographies

This section extends conventional homographies to include the degenerate cases by considering arbitrary - including eventually singular - endomorphisms (i.e. linear maps on a vector space) acting on configurations of points. In particular, regular transformations up to scale of a 3-dimensional vector space belong to the group of homographies which is an open subset of the projective space . Its complementary is the set of singular endomorphisms up to scale, a cubic hypersurface defined by for and containing the fundamental subvariety and the essential manifold .

Planar homographies represent regular transformations between two projective planes of 2D views. Thus, any homography is an element of the projectivized linear group , where is the general linear group acting on the projective model of each view. Given a reference for , each element of can be represented by a regular matrix, i.e. with non-vanishing determinant. By construction, homographies (regular transformations up to scale) can not include degenerate transformations such those appearing in fundamental or essential matrices. Then, these matrices can be considered as “degenerate” endomorphisms (represented by defficient rank matrices up to scale) of an abstract real 3D space with .

Fundamental matrices are defined by degenerate bilinear forms linking pairs of corresponding points. The set of pairs of corresponding points is called the join of two copies of . This join is defined by the image of the Segre embedding giving a four-dimensional variety of that determines the 7D subvariety of singular endomorphisms up to scale. Addition of the singular cases “completes” the homographies (regular transformations), treating fundamental and essential matrices as degenerate transformation between two projective planes inside the set of a completion or homogeneous endomorphisms.

To understand how transformations can be extended from a geometric to a kinematic framework, it is convenient to introduce the differential approach for the regular subset. In terms of algebraic transformations, one mus replace the Lie group of regular transformations by its Lie algebra where is the neutral element of (the identity matrix for matrix groups); in particular, . As usual in Lie theory, denote the elements of the group , and the elements of its Lie algebra . In particular, any endomorphism can be described by a matrix representing a point up to scale.

The exponential map is a local diffeomorphism (with the logarithm as inverse) that can be applied to degenerate matrices for . In general, the set of homographies is an open set of where , whose complementary is given by the algebraic variety of degenerate matrices, i.e. , where is the determinant of . We are interested in a better understanding of degeneracy arguments from the analysis of pencils (in fact tangent directions) passing through a lower rank endomorphism. The “moral” consists of the following simple remark: the original action given by a matrix product, induces an action on linear subspaces by means the -exterior power of the original action. Next paragraph illustrates this idea with a simple example.

In particular, a line represents a pencil (uni-parameter family) of endomorphisms , i.e. a linear trajectory in the ambient space where . A general line has degenerate endomorphisms corresponding to the intersection denoted by . Inversely, the generic element of the linear pencil for is a homography away from the variety of degenerate endomorphisms5.

For , the intersection of a general projective line with the cubic algebraic variety defined by gives generically three different degenerate endomorphisms. In particular, if is tangent to at least two elements of can coalesce. An ordinary tangency condition is represented by , where (resp. ) represents a tangency (resp. simple) contact point corresponding to the intersection of with . Linear pencils of matrices representing endomorphisms are interpreted as secant lines in the projective ambient space.

In general, -secant varieties to a variety are defined by the set of points lying in the closure of -dimensional subspaces generated by -tuples of affine independent points generating linearly independent vectors. They can be formally constructed by using the -th exterior power of the underlying vector space that allows to manage -tuples of points for . This statement can be adapted to the underlying vector space of the Lie algebra with its natural stratification by the rank of any classical group . The locally symmetric structure is the key for extending the concept in the presence of singularities. Although this construction is general for , it can be constrained to manage eventually degenerate tensors. Actually, this approach allows to connect old based-perspective methods using homographies with tensor-based methods.

### 3.1 Fundamental variety

This subsection highlights the geometry of subvarieties parameterizing rank deficient endomorphisms (up to scale) for a three-dimensional vector space .

The graph of a planar homography is given by the set of pairs of corresponding points contained in two views modeled as projective planes fulfilling . From a global point of view, the ambient space is given as the image of the Segre embedding , i.e. it is a -dimensional algebraic variety given at each point by the intersection of four functionally independent quadricsfaugeras1993three. As parameterizes the set of bilinear relations between two projective planes, and each projective plane has a projective reference given by points, a general homography can be described in terms of two -tuples of points that can be re-interpreted as the eight (nine up to scale) projective parameters of a general matrix .

The space of -matrices up to scale is a projective space , which is homogeneous by the action of the projective linear group . The projective linear group induces an action on that breaks the initial homogeneity of due to the rank stratification given by three orbits. Each orbit is characterized by the rank constancy of a representative matrix. In particular, if denotes the algebraic subvariety of matrices (up to scale) of rank , then there is a natural rank stratification . Planar homographies may be viewed as elements of up to scale. This rank decomposition is restricted in a natural way to the fundamental variety .

#### Canonical rank stratification

This subsection includes some results regarding the endomorphisms of a vector space , where denotes the rank of a generic element. The first result provides a description of endomorphisms and its singular locus corresponding to degenerate endomorphisms . The second result gives its structure as a locally homogeneous space, i.e. as a disjoint union of -orbits by the action of on the vector space of . As usual, their elements are regular or eventually degenerate matrices, but their meaning is different as Lie group or Lie algebra, respectively.

###### Proposition 1

For any three-dimensional vector space : a) the set of singular endomorphisms is a algebraic variety of codimension given by a cubic hypersurface for , which is a subregular orbit by the action of corresponding to “subregular” elements located in the adherence of the set of homographies in ; b) its singular locus is given by rank degenerate endomorphisms , which is a codimension manifold (smooth subvariety) diffeomorphically equivalent to

###### Proof

a) It is proved taking into account the characterization of singular endomorphisms by the vanishing of the determinant of a generic matrix. b) By taking the gradient field in , its singular locus is locally described by the vanishing of determinants of all minors of a generic matrix representing any endomorphism up to scale. Using as local coordinates in , if , then a local system of independent equations (local generators for the ideal of the determinantal variety representing rank 1 matrices) is locally given by

 a11−a01a10a12−a02a10a21−a01a20a22−a02a20 (1)

in the open coordinate set of . They are functionally independent (i.e. its jacobian matrix has maximal rank) between them. Hence, they define a smooth variety of codimension 4 (differential map of the above equations has maximal rank), which is locally diffeomorphic to . The induced group action allows to extend the local diffeomorphism to a global diffeomorphism. In particular, it is locally parameterized by , , , corresponding to elements in the complementary box of (obtained by eliminating the row and the column of ).

Formally, the involution on spaces that exchanges subindexes (fixed points for transposition) leaves invariant the first and fourth generators, and identifies the second and third generators between them. Such involution corresponds to a representation of the symmetric group, giving the local generators for the Veronese variety of double lines, which is isomorphic to the dual counted twice.

Anyway, the rank stratification can be reformulated in homogeneous coordinates as follows:

###### Corollary 1

The action of on gives an equivariant decomposition in three orbits characterized by the rank of the representative matrix up to scale. In particular: 1) the set of rank 1 endomorphisms (up to scale) is a 4D smooth manifold whose projectivization is diffeomorphic to , which is a closed orbit by the induced action; 2) the set of rank 2 endomorphisms (up to scale) is a 7D subregular orbit; and 3) the set of homographies corresponding to regular endomorphisms (up to scale) is the regular orbit.

The stratification of endomorphisms up to scale involving the projective model of planar views can be geometrically reinterpreted by reconstructing the variety of degenerate endomorphisms as the secant variety in of the smooth manifold . Secant varieties are explained in Section 3.1.2.

The action of can be extended in a natural way to the -th exterior power involving -tuples of vectors and their transformations for . Thus, a locally symmetric structure is obtained for arbitrary configurations of -tuples of vectors (or -tuples of points). It is extended in a natural way to linear envelopes of -dimensional vector subspaces or, in the homogeneous case, to -dimensional projective subspaces giving linear envelopes for any geometric object contained in the ambient space.

The set of -dimensional linear subspaces are elements of a Grassmann manifold ; its projective version is denoted as . Grassmann manifolds are a natural extension of projective spaces. They also provide non-trivial “examples” for homogeneous spaces and their generalization to symmetric spaces or spherical varieties, jointly with superimposed universal structures (fiber bundles). They have been overlooked over the years despite the presence of the analysis based on subspaces in a lot of tasks. A brief introduction to Grassmannian manifolds and their applications is provided by ye2016schubert.

#### Secant varieties

Homographies, fundamental or essential matrices can be viewed as PL-uniparametric families (linear pencils) of matrices that can be represented by secant lines. Similarly, secant planes would correspond to PL-biparametric families (linear nets) of matrices, and so on. Additional formalism is required for a systematic treatment of these families.

Secant varieties provide a PL-approach to any variety relative to any immersion . They are given as the closure of points where with are linearly independent. The -th secant map associates to each collection of l.i. points their linear span . The closure of the graph is called the secant incidence variety. The projection of the last component on the Grassmann manifold is called the -th secant variety (of secant -dimensional subspaces) of and it is denoted by as subvariety of .

Since ordinary incidence conditions are invariant by the action of the projective group, secant incidence varieties represent projectively invariant conditions too. These conditions extend the well-known tangency conditions . Next, we provide a classical definition for smooth manifolds:

###### Definition 1

For any embedding of a connected regular -dimensional manifold , the secant variety (also called “chordal variety” in the old terminology) is defined by the closure of points lying on lines (called “chords”, also), where are different points belonging to , where is the diagonal of .

Obviously, if the secant variety fills out the ambient projective space. More generally, the following result is true:

###### Lemma 1

If is a -dimensional smooth connected manifold, the expected dimension of is equal to .

The lemma is a consequence of a computation of parameters on connected smooth varieties. The dimension of the secant variety can be lower, but exceptions are well-known for a specific type of low-dimensional varieties called Severi varietieszak1986severi. In particular, the chordal variety of the -dimensional Veronese variety has dimension , instead of the expected dimension , providing the first non-trivial example of a Severi variety. More generally, if is a -dimensional connected manifold and , as is a connected variety, then (see (shafarevich2013basic, Page 40) for more details about the Veronese embedding).

An alternative description for a secant variety can be provided in a purely topological way. Let define the diagonal of the product as , i.e. the set of pairs such that . If is a smooth -dimensional variety, then is diffeomorphic to through the diagonal embedding. Hence, the normal bundle is isomorphic to the tangent bundle . Note that . This topological description is useful to detect “regular” directions thorugh the singular locus:

Each pair can be mapped to the line (also denoted as ). This map defines a morphism called the secant map of lines. The closure of its image contains the set of tangent lines to , which correspond to limit positions of secant lines when coalesce in one point .

The closure of the graph of the secant map of lines is a -dimensional incidence smooth projective variety in the product whose projection on last component is, by definition, the -secant -dimensional subvariety of . The tangent space at each point can be described as the set of tangent lines to curves having a contact of order 2 with . This definition is formally extended to the singular case by taking local derivations. However, we use a more simplistic approach based on the continuity arguments and a simple computation of parameters, which gives the following result:

###### Proposition 2

The incidence variety of secant lines contains the -dimensional space of the tangent bundle of the manifold . Its projection

 S1(M)⊂Grass1(PN)

on the image of by the secant map is a -dimensional subvariety of the Grasmannian of lines.

These descriptions show how secant lines can be understood in terms of the geometry of the ambient projective space or, alternately, in terms of the geometry of Grassmannians of lines . The arguments are extended to higher dimension and singular varieties in fulton1984intersection. Furthermore, they correspond to decomposable tensors which are useful for estimation issues, also.

#### Secant to degenerate fundamental matrices

Results in previous subsection allow to manage degeneracies in regular transformations and to perform a PL-control in terms of limit positions of secant varieties. Its extension to singular cases requires also the following result:

###### Proposition 3

Let denote the three-dimensional algebraic variety of degenerate rank fundamental matrices as , then and .

###### Proof

It suffices to prove that if , then , i.e. . So, let define , where is the -th column of the matrix for . Then, is computed as the arithmetic sum (up to sign) of determinants which are always null. More explicitly,

 ∣F+λF′∣ =∣f1 f2 f3∣ + λ(∣f1 f2 f′3∣+∣f1 f′2 f3∣+∣f1 f2 f3∣) + λ2(∣f1 f′2 f′3∣+∣f′1 f2 f3∣+∣f′1 f′2 f3∣) + λ3∣f′1 f′2 f′3∣

The first and last summand vanish since . By developing each determinant by the elements of the column (Laplace) and by using that all -minors of and vanish the proposition is proved.

This result can be also applied to symmetric matrices up to scale in the projective space of plane conics:

###### Corollary 2

Let denote the two-dimensional algebraic variety of degenerate rank symmetric matrices as , then and .

###### Proof

The second equation is trivial by connectedness properties and dimensional reasons. An intuitive proof for the first equation is obtained from Proposition 3 by the involution that exchanges subindex coordinates. Intuitively, the generic element of any pencil generated by two double lines is a pair of intersecting lineszak1986severi.

The extension of the Corollary 2 to the space of quadrics in is meaningful for 3D reconstruction issues. Indeed, a projective compactification of essential manifold is isomorphic to a hyperplane section of the variety of quadrics in of rank (see below), whose elements are pairs of planes (eventually coincident).

Nevertheless their simplicity, these results are useful because provide a general strategy to manage degeneracies. In particular, for each degenerate fundamental matrix , a generic segment connecting two rank 1 fundamental matrices gives a generic rank 2 fundamental matrix. As consequence, a generic perturbation with any PL-path removes the indeterminacy, and recovers a generic rank 2 fundamental matrix. This perturbation method (valid for stratifications with “good incidence properties” for adjacent strata) provides a structural connection between fundamental matrices and homographies, which can be extended to essential matrices (see Section 4.1.4).

### 3.2 Removing indeterminacies

There are different kinds of indeterminacies for multilinear approaches including structural relations between pairs (given by fundamental and essential matrices, typically) or triplets of views (trifocal tensors, e.g.). All of them can be interpreted as singularities of the variety of corresponding tensors, including the fundamental variety or the essential variety of . Indeterminacies can be removed using secant lines to the singular locus of (introduced in Section 3.1) or more generally -th exterior powers.

Secant lines allows to control degenerated matrices using chords cutting singular varieties. These chords interpolate between more degenerate cases to recover a valid case. This subsection explains an alternative recovery strategy based on the “recent history” of tangent vectors to the camera poses. This also requires a careful analysis of tangent spaces to matrices in terms of their adjoint matrices.

#### A naive approach

A regular transformation can be described by a matrix with non-vanishing determinant or, from a dual viewpoint, by the adjoint matrix . Entries of the adjoint matrix are the adjoint of each element up to scale; in the regular case, one must multiply with . Hence, entries of the adjoint matrix represent up to scale the gradient field of the array linked to .

For non-regular matrices (i.e. matrices with vanishing determinant) both descriptions are no longer equivalent between them. Let study the simplest case for where the adjoint can be extended to non-regular matrices. In the regular case, the adjoint is formally described as the second exterior power of . The action of on induces an action of the second exterior power on the space of 2-dimensional subspaces (bivectors), whose projectivization represents lines of the projective plane associated to the image plane. Thus, the Adjoint map replaces the study of loci characterized by features by their dual features supported by projective lines. Loci and enveloping hyperplanes are equivalent between them for regular matrices.

This naive approach has some implications to remove indeterminacy when . In order to understand them must be replaced by an enlarged space that includes the different ways of approaching each element by an “exceptional divisor”. Each divisor is supported by a finite collection of hypersurfaces in the ambient space representing different approaching ways to the singular locus, including elements of . This process is known in Algebraic Geometry as a blowing-up or -process shafarevich2013basic.

An almost-trivial example Bilinear maps on a vector space are represented by a -dimensional arbitrary space coupled with an inner product . A direct computation shows that an orthogonal basis with and is given by

 (1000),(0001),(0110),(01−10)

Note that and are second order nilpotent operators and that the last generator provides the bilinear structural constraint for the symplectic geometry on libermann2012symplectic; guillemin1990symplectic. In this case, , , , and , with , and .

From a projective viewpoint, the set of bilinear maps on the projective line can be reinterpreted in terms of the image of the Segre embedding given by

 ([x1:x2],[y1,y2])↦[x1y1:x1y2:x2y1:x2y2]

Let be then defines a one-sheet hyperboloid in . Hence, the set of bilinear maps on (up to scale) can be modelled with such hyperboloid. The main novelty here concerns the ruling given by the group action on the projective lines and . In this case, the adjoint map induces an involution that can be translated to a complex conjugation between the generators for each ruling.

#### A more formal approach

From a local topological viewpoint, degenerate endomorphisms can be studied in terms of limits of tangent vectors to curves through . Such curves represent trajectories in the matrix space connecting “consecutive” poses for a camera. Regularity of generic points of such curves allows to define tangent vectors at isolated degeneracies by means secant lines.

A more intrinsic approach to tangency conditions must include the dual matrix for any . It is defined at each point by the adjoint matrix whose entries are the signed determinants of complementary minors of . If is a regular matrix, then is a power of , i.e. a unit from a projective viewpoint. We are interested in extending this construction to singular endomorphisms by using intermediate exterior powers. Their closure in the corresponding projective space forms the variety of complete endomorphisms.

To begin with, a first order complete endomorphism representing a degenerate planar transformation is given, up to conjugation, by a pair of matrices , where is an endomorphism of a 3D vector space , and represents its dual given up to scale by the adjoint matrix . The replacement of a matrix with its adjoint transforms any incidence condition (pass through a point for a conic, e.g.) into a tangency condition (dual line becomes tangent at a point, e.g.). Moreover, this exchange between projectively invariant conditions does not depend of the dimension. For regular homographies, there exists a natural duality between descriptions in terms of the original matrix and its adjoint matrix , giving the natural duality between incidence and tangency conditions for smooth “objects” (endomorphisms, in our case). Hence, the only novelty appears linked to singular strata which can be illustrated by its application to the variety of singular fundamental matrices when . The dual construction is compatible with the induced action of on and its restriction to . Let , then .

For arbitrary dimension, the entries of the adjoint matrix for are interpreted as determinants corresponding to the components of . The next iteration for gives the determinants of -minors as generators for the ideal of . Their vanishing defines the singular locus of the variety . Symbolically, can be formulated as a double iteration of the adjoint map generated by the vanishing of determinants of minors of size of . These determinants are the generators of , which is the dual of .

The description of the previous paragraph can be geometrically reinterpreted in terms of arbitrary codimension subspaces. In particular, the extension of the adjoint map can be algebraically interpreted as a gradient field. For any endomorphism let consider the -tuples

 (X,2⋀X,…,n⋀X) , (2)

where is the -th exterior power of , whose entries are given by the determinants of the -minors of . It is an element of the exterior algebra defined by the direct sum of exterior powers of . The iteration of the gradient field given by the determinant function as can be interpreted as “successive derivatives” on the space of endomorphisms. Let us remark that traces of exterior powers are the coefficients of the charatceristic polynomila ;, which can be reinterpreted (in the complex case) in terms of eigenvalues. Thus, in this case all the information is computable in terms of SVD with usual interpretation for the ordered collection of eigenvalues.

In arbitrary dimension, the generic case corresponds to the regular orbit, i.e. endomorphisms with (automorphisms). By iterating the construction of exterior powers, one can associate an algebraic invariant given by the multirank . Looking at Figure 2, the regular case for (resp. ) corresponds to bilinear forms with birank (resp. the multirank with self-duality for the mid term), which can be reinterpreted in terms of quadratic forms. The case for non-regular orbits is constructed recurrently: let be the dimension of , the indeterminacy is removed by adding the complete bilinear as the linked quadratic forms on .

For example, for any symmetric endomorphism whose projectivization is a rank plane conic, there exists a double line whose kernel is the whole line. The reduced kernel is also the support for a conic on the line given by two different points (rank 2) or a double point (rank 1), which define two orbits labeled as and in Figure 2. Similarly, for a rank quadric supported by a double plane , the kernel is the whole plane that supports an embedded complete conic in the double plane with biranks , , and . Hence, the most degenerate orbits of complete quadrics have multiranks , , and . If the exterior powers represent geometrically the matrices acting on envelopes by -dimensional tangent linear subspaces to “any object” contained in of increasing dimension. In this case, (a) the action of induces a conjugation action of on itself; (b) the -th exterior powers of can be considered as elements of the -th exterior power of .

Essential manifold of regular essential matrices is embedded in , a -dimensional projective space. The extension of complete homographies on to creates complete collineations . The third component is in fact the adjoint matrix of . This construction provides a general framework to obtain compactifications (as complete varieties) of orbifolds corresponding to and as degenerate endomorphisms.

The basic idea for extending regular to singular cases is based on adding infinitesimal information from successive adjoint maps, which is interpreted as the iteration of the gradient operator applied to the determinant of square submatrices. For generic singularities (i.e. for corank ) it suffices to replace the original formulation by its dual, which gives the tangent vector for small displacements. For singularities with corank successive exterior powers and complete endomorphisms must be considered. This differential description allows to interpret complete endomorphisms in terms of the “recent history” along the trajectory. In order to simplify the developments we consider the particular case .

#### A symbolic representation

Adjacency relations between closures of orbits for rank stratification of can be symbolically represented by the oriented graph of Figure 1. The vertices of the triangle represent an orbit labeled with , according to the rank of the matrix representing the endomorphism. Oriented edges are denoted as , and and represent the following degeneracies:

• , corresponding to degeneracies of endomorphisms to fundamental matrices.

• , corresponding to degeneracies from fundamental to degenerate fundamental matrices.

• , corresponding to degeneracies from rank endomorphisms to degenerate fundamental matrices

Right-side vertices of Figure 1 represent the biranks corresponding to the original matrix and its generalized adjoint. A simple example is useful to illustrate the idea. Let suppose as a diagonalizable matrix and and as the eigenvalues of , then fundamental matrices have at least one vanishing eigenvalue. In the complex case canonical diagonal forms would be equivalent to diagonal matrices whose adjoint would be , , and , respectively. The adjoint of the third type is identically null so, initially it does not provide additional information about the most degenerate case. Information about regular elements in the degenerate support corresponding to neighbor tangent directions (infinitesimal neighborhood) must be added to avoid the indeterminacy. must be replaced by the exceptional divisor of the blowing-up of with center the smooth manifold . has the orbits by the adjoint action as components, represented by the pairs and .

The blow-up of the graph replaces the oriented triangle whose vertices are labeled as and with a new graph . This graph is an oriented square whose vertices are labeled as , and . These labels represent the biranks of pairs of endomorphisms . Biranks are relative to the generic elements of the topological closure for the orbits of the product action , where , on the space , where , acting on the graph of the adjoint map.

Hence, biranks encapsulate numerical invariants for the natural extension of the action of the Lie algebra corresponding to , using pairs of infinitesimal transformations on the original and dual spaces. The simultaneous management of complete endomorphisms is more suitable from the computational viewpoint. Indeed, a simple extension of SVD methods to pairs of endomorphisms allows an estimation of the generators of Lie algebras and its dual easier than the estimation of the generators for the original Lie group .

### 3.3 Complete fundamental matrices

This section adapts constructions of endomorphisms in any dimension shown in precedent section to fundamental matrices. The notion of complete fundamental matrix is crucial to perform a control of degenerate cases from a quasi-static approach. This means that we are not taken into account the kinematics of the camera. Nevertheless, as the adjoint represents the gradient vector field at the element , there is a measurement of “local variation” around each element .

###### Definition 2

The set of complete fundamental matrices is the closure of pairs where is a fundamental matrix of rank 2, and is its adjoint matrix whose entries are given by the determinants of all -minors of .

Degeneracies of endomorphisms up to scale linked to camera poses can be controlled with the insertion of tangential information and the restriction of the construction to . This information allows to represent “instantaneous” directions in terms of all possible directional derivatives represented by the adjoint matrix. Thus, despite and consequently , the adjoint matrix provides a parameterized support (in fact a variety) to represent “directions” along which the degeneracy occurs. However, this remark is not longer true for the most degenerate cases corresponding to rank matrices that require a set of pairs of degenerate fundamental matrices.

#### Complete pairs of fundamental matrices

In the absence of perspective models supporting arbitrary homographies, fundamental matrices are used to avoid indeterminacies. For consecutive camera poses, two fundamental matrices (for views and ), and (for views and ) provide a structural relation between both views.

The dual construction corresponding to the gradient at each point can be restricted just to rank 2 fundamental matrices. This provides a tangential description with information about the first order evolution of according to tangential constraints. From a more practical viewpoint, tangential information can be approached by secant lines in a PL-approach that connects rank 2 fundamental matrices . However, this construction becomes ill-defined when since the adjoint map is identically null. Then, tangential directions corresponding to endomorphisms of are added to avoid this indeterminacy.

For regular matrices the extended approach for complete objects is compatible with the differential approach given by a smooth interpolation along a geodesic path connecting consecutive complete fundamental matrices and . These are obtained as the lift of a geodesic path to the tangent space. From a theoretical viewpoint, lifting is performed by restricting the logarithm map. In practice, secant lines provide a first order approach to geodesics.

A more detailed study of the geometry of degenerate fundamental matrices is required to recover a well-defined limit of tangent spaces in the singular case. This study must include procedures for selecting a PL-path (supported on chords) connecting the degenerated with its neighboring generic fundamental matrices. In practice, if the sampling rate is high enough there will be no meaningful difference between pairs of matrices. This generates uncertainty about the direction to approach the tangent vector. It can be solved with a coarser sampling rate along the “precedent story”.

#### Incidence varieties and canonical bundle

A local neighborhood of relative to is the unit sphere corresponding to the fiber of the punctured normal bundle of in . Normal bundles are given by a quotient of tangent bundles. Hence, tangent bundles of and must be computed. The latter can be described as the blow-up of with center in the diagonal . This diagonal is isomorphic to the smooth manifold so its tangent bundle is also isomorphic to the tangent bundle of . Hence, it suffices to compute the latter and reinterpret it in geometric terms.

Note that is a smooth manifold diffeomorphic to , so their tangent bundles are isomorphic. Thus, can be reinterpreted in terms of incidence varieties. Simplest incidence varieties in the projective plane are given by .

There are two projections on and to interpret the incidence variety as the canonical bundle of the projective plane. This elementary construction is extended to subspaces of any dimension with the canonical bundle on the Grassmannian of -dimensional subspaces. An example is the incidence variety , where denotes the Grassmannian of lines in .

The restriction to the fundamental variety of the second projection on gives the secant variety to that fills out all the ambient projective space. In other words, along each point pass a secant line to . The same is also true for instead of . Furthermore, an interesting result can be formulated using the same notation:

###### Proposition 4

The secant variety is a generically triple covering of that ramifies along the -dimensional tangent variety (total space of the tangent bundle) corresponding to tangent lines having a double contact at each element .

#### Triplets of fundamental matrices

This Section is intended to provide an algebraic visualization of neighboring matrices at the most degenerate case, which avoids the indeterminacy at rank elements. We use a geometric interpretation of the blowing-up process explained in shafarevich2013basic in terms of secant varieties.

In our case, the blowing-up of the variety replaces each rank degenerate fundamental matrix with a 3D subspace generated by four linearly independent vectors. They can be interpreted in terms of secant lines connecting the point for matrix with four independent points belonging to the subregular orbit . Hence, each extended face of a generic “tetrahedral configuration” (see Figure 2) represents a 3D secant projective space to displaying degeneracies at each .

The cubic hypersurface representing fundamental matrices in is not a ruled variety. Thus, a generic triplet of fundamental matrices generates a -dimensional secant plane to the hypersurface . Obviously, the variety of secant lines to and trisecant 2-planes to fills out the projective space of endomorphisms up to scale. More specifically, any homography can be expressed as a linear combination of three generic fundamental matrices (similarly for essential matrices).

The nearest -secant -plane for each rank fundamental matrix can be computed using a metric on the Grassmannian of -planes. The most common metric is the inner product

 =tr(X.Y)

of . In particular, at each degenerate endomorphism corresponding to a fundamental matrix (rep. essential matrix ), two different eigenvalues (resp. a non-zero double eigenvalue ) are obtained for .

In practice, the direction to choose as “escape path” should correspond to the nearest with maximal distance in the plane () of non-null eigenvalues. This distance is determined w.r.t. the eigenvalues or of the degenerate fundamental matrix (similarly for the essential matrix). The application of this theoretical remark would must allow to escape from degenerate situations to avoid collisions against planar surfaces (corresponding to walls, floor, ground, e.g.) whose elements do not impose linearly independent conditions to determine . From a more practical viewpoint, the problem is the design of a control device able of identifying the “best” escape path in a continuous way, i.e. without applying switching procedures. In the next paragraph we give some insight about this issue.

## 4 Extending the algebraic approach

This section explains how complete matrices, which can be read in terms of successive envelopes, provide specific control mechanisms to avoid degeneracies appearing in rank fundamental or essential matrices.

A basic strategy to analyze and solve the indeterminacy locus of an endomorphism consists in augmenting the original endomorphism by their successive exterior powers. The properties of the adjoint matrix provide a geometric interpretation in terms of successive generalized secant envelopes by -dimensional subspaces6. Consecutive iteration on the adjoints can be viewed as a Taylor development so that when decreases, all complete objects contained in can be added to remove degeneracies.

The lifting of the action of on to their -th exterior powers delivers a structure as locally symmetric spaces for the set of complete objects linked to . Here can be replaced with a group , the dual of (corresponding to take adjoint matrices) with and the induced action by the adjoint map giving the adjoint action . Then, the -th exterior power is the natural extension of the adjoint map. This map induces the corresponding -th adjoint action of on that extends the original action of on . This simple construction is applicable for all actions of classical groups to remove their possible indeterminacies on Lie algebras.

The simplest non-trivial example is the pair , which defines a space of dimension for . In this case, the regular matrices of (automorphisms of ) act on the arbitrary matrices of (endomorphisms of ). In practice it requires to compute the determinants of -minors of a regular matrix acting on -minors of a arbitrary matrix. This explains the jump from original ranks of arbitrary -matrices and their second powers to ranks .

An adaptation of the general linear approach to the euclidean case provides a decomposition of the -matrices in -blocks that can be reinterpreted in terms of ordinary rotations. In addition, this construction can be adapted to bilateral (product or contact) actions in terms of double conjugacy classes. We are interested in the locally symmetric structure of the rank-stratified set of projection matrices linking scene and views models. A simple description of this structure for any space allows to propagate control strategies by using local symmetries, without using differential methods, no longer valid in the presence of singularities.

The completion of planar homographies in to include fundamental matrices can be extended to any dimension and re-interpreted in matrix terms. To achieve this goal, it is required to consider the projectivization of a -dimensional space , to construct the rank stratification of matrices in and to take the -th exterior power (up to scale) of them for . The locally symmetric structure for the resulting completed space is obtained by the induced action of on the -th exterior power of . This structure justifies positional arguments for minimal collections of corresponding elements (points, lines, or more generally, linear subspaces) and controls their possible degeneracies in terms of adjacent orbits.

The construction of the above completion poses some challenges. For example, its topological description in for requires additional -dimensional linear subspaces in the ambient projective space . Using contact constraints for linear subspaces has an equivalence to scene objects in terms of PL-envelopes by successive higher dimensional linear subspaces . Due to space limitations, we constrain ourselves to the case and the completion of essential matrices.

The fact that (including degeneracies) is the Lie algebra of (only regular transformations) links algebraic with differential aspects. Hence, exponential and logarithm maps provide a natural relation between both of them. However, as fundamental and essential matrices play a similar role for affine and euclidean frameworks (as non-degenerate bilinear relations), a common framework where both interpretations are compatible is required for an unified treatment of degenerated cases.

### 4.1 Essential manifold

The essential constraint for pairs of corresponding points from a calibrated camera is given by . The set of essential matrices is globally characterized as a determinantal variety in (kileel2017algebraic, Section 2.2). The author explores the structure of as a locally symmetric variety and a completion (not necessarily unique) obtained using elementary properties of adjoint matrices.

Any ordinary essential matrix has a decomposition , where is a rotation matrix and is the skew-symmetric matrix of a translation vector . Essential and fundamental matrices are related through where (resp. ) is an affine transformation acting on right (resp. left) on the source (resp. target space). In algebraic terms, they belong to the same double conjugacy class by the diagonal of the -action of two copies of the affine group, where is the direct product of right and left actions. Hence, essential matrices can be considered as equivalence classes of fundamental matrices. The following result gives a synthesis of the above considerations:

###### Proposition 5

The variety of extended essential matrices of rank is a quotient of the variety of extended fundamental matrices of rank . More generally, the stratified map is an equivariant fibration between stratified analytic varieties for natural rank stratifications in .

The exchange between projective, affine and euclidean information and the analysis of degenerate situations requires a general framework where rank transitions can be controlled in simple terms. To accomplish this goal, a locally symmetric framework that represents degenerate cases must be developed.

Degeneracies can be studied in the space of multilinear relations between corresponding points, from which the essential matrix is estimated kim2010degeneracy. However, these estimations are performed in the space of configurations of points without considering the degeneracies of the matrices in the space of endomorphisms arising from an algebraic viewpoint. The secant line connecting two degenerate endomorphisms is translated in linear parameters in the same way as in kim2010degeneracy. Additional constraints relative to fundamental or essential matrices help to reduce the number of parameters.

If is a variety with singular locus , the regular locus is denoted as . In particular, the set of regular fundamental matrices is denoted by and the set of essential matrices as . Rank stratification of is (resp. ), where the subindex denotes the algebraic subvariety of matrices with rank , up to scale 7.

Two global algebraic and differential results to consider are the following ones:

• The essential variety is a -dimensional degree subvariety of , which is isomorphic to a hyperplane section of the variety of complex symmetric matrices of rank floystad2018chow. The singular locus is isomorphic to via the degree Veronese embedding whose image is the variety of double planes. In particular .

• If denotes the total space of the tangent bundle of , then the essential regular manifold is isomorphic to the total space of unit tangent bundle of given by .

Nevertheless their local description in terms of Lie algebras, all the above isomorphisms are global since any Lie group is a parallelizable manifold. If is connected, the isomorphisms are infinitesimally given by the translation . In the euclidean framework, this description allows to decouple rotations and translations. The rest of this Section explains local and global properties of extended and their relations with fundamental matrices.

#### Parameterizing the essential manifold

An essential matrix is a -rank matrix with two equal eigenvalues and a diagonal form , which in the complex case is projectively equivalent (up to scale) to . Singular Value Decomposition (SVD) decomposes in a product where are orthogonal and . The sign of the determinant can be chosen without modifying the SVD. Then, the fibration given by is a submersion with a -dimensional kernel representing the ambiguity for choosing the basis of the space generated by the first two columns of and .

The description of in terms of the fibration allows to decompose any element in “horizontal” and “vertical” components for the tangent space to the product . This is crucial to reinterpret the decomposition in locally symmetric terms and to bound errors linked to large baselinessubbarao2008robust.

#### A differential approach

The set of regular essential matrices is an open manifold that can be globally described in terms of the unit tangent bundle to the special orthogonal group representing spatial rotationsma2001optimization. Also, is the Lie algebra of , i.e. the vector space of skew-symmetric -matrices that can be interpreted as translations in the tangent plane at each . Then, is a -dimensional algebraic variety contained in the total space of the unit tangent bundle . In this bundle each fiber takes only unit tangent vectors so we restrict to unit vectors .

The isomorphism enables a reinterpretation of the unit vectors as spatial rotations modulo planar rotations. Each element determines a unique rotation axis, where the rotation through angles and are identical. Hence, is homeomorphic to , which provides a general framework for a projective interpretation in terms of space lines (see (ma2001optimization, Section 3.2)). Inversely, euclidean reduction of projective information can be viewed as a group reduction to fix the absolute quadric that plays the role of non-degenerate metric (see Section 4.1.3).

#### A local homogeneous description

The projective ambiguity of projective lines as rotation axis gives two possible solutions for corresponding elements of (dubbelman2012manifold, Section 3). This ambiguity can be modelled as a reflection that exchanges the current phase with the opposite phase between them. Hence, pairs of regular essential matrices in , where the the second component is generated by the logarithmic map, generate a quadruple ambiguity corresponding to two simultaneous reflections.

Using the topological equivalence between and , the ambiguity can be represented by the product of two copies of the tangent bundle of the projective space where elements and are identified by the antipodal map. This natural identification of the tangent bundle has not a kinematic meaning from the viewpoint of the “recent history”. Hence, in order to solve the ambiguity, a -constraint must be inserted into the essential matrices for precedent camera poses. However, this constraint is only valid under non-degeneracy conditions for essential matrices, i.e. the eigenvalue must be non-null. Otherwise, essential matrix “vanishes” and it cannot be recovered.

#### A remark for matrix representation

The manifold of regular essential matrices can be visualized as a smooth submanifold of spatial homographies given as the open set of regular transformations as points of . These regular transformations are described by automorphisms of a -dimensional vector space whose Lie algebra is given by , including possible degeneracies. Our aim is to study these degenerate cases by using locally symmetric properties extending . Metric distortions must be avoided when approaching to the singular locus.

The vector space of -matrices has a natural stratification by the rank denoted by . Here, represents the algebraic variety defined by the vanishing of all determinants of size