Geometrical analysis of polynomial lens distortion models††thanks: This work has been partially supported by the Ministerio de Economía, Industria y Competitividad (AEI/FEDER) of the Spanish Government under projects TEC2013-48453 (MR-UHDTV), TEC2016-75981 (IVME) and TIN2013-47630- C2-2- R (SPACES-UPM).
Polynomial functions are a usual choice to model the nonlinearity of lenses. Typically, these models are obtained through physical analysis of the lens system or on purely empirical grounds. The aim of this work is to facilitate an alternative approach to the selection or design of these models based on establishing a priori the desired geometrical properties of the distortion functions. With this purpose we obtain all the possible isotropic linear models and also those that are formed by functions with symmetry with respect to some axis. In this way, the classical models (decentering, thin prism distortion) are found to be particular instances of the family of models found by geometric considerations. These results allow to find generalizations of the most usually employed models while preserving the desired geometrical properties. Our results also provide a better understanding of the geometric properties of the models employed in the most usual computer vision software libraries.
Keywords:Lens distortion Camera calibration Polynomial model
The correction of lens distortion is a relevant problem in computer vision and photogrammetry Hartley-Zisserman (). Lens distortion models the departure of the image capturing device from the theoretical pin-hole model and consists essentially in an image warping process.
Most of the proposed lens distortion models are given by an analytical expression of the space variables and the model parameters, although some efforts have also being made in order to depart from concrete analytical expressions Hartley (); Ricolfe2 (). These closed-form expressions usually express the position of the distorted points as a function of the ideal undistorted points given by the pinhole assumption, although in some cases it is the inverse of this function what is given by the model functions Claus ().
Lens distortion models can either result from the analysis of the physical problem or from a pragmatic approach led by the empirical capacity of the model to fit the observed data and the existence of practical algorithms to compute the model parameters. The concrete parameters of the distortion function are frequently computed within the bundle-adjustment process of a 3D scene reconstruction Claus (); Weng (); Li (), but it is often possible to obtain these parameters from a single image that contains an element of known geometry, such as a calibration grid or a set of lines Alvarez (); Ma3 (); Strand (); Wu (); Devernay ().
The first and probably most employed analytical form of lens distortion models is given by polynomials Conrady (); Brown (); Weng (). A natural generalization is that of rational functions Claus (), although some empirical studies Tang () attribute a similar modeling capabilities to both approaches.
A large part of the literature on these models assumes a radial rotationally invariant (RRI) distortion function (Hartley-Zisserman, , p. 191). This strong geometrical requirement stems from the assumption that the capturing system is a rotationally symmetric structure. While these models suffice for some applications, those requiring higher precision must also account for such phenomenons as the non-alignment of the axes of the lens surfaces or the lack of paralellism of the lens and the imaging surface. The first is usually addressed by the decentering lens distortion model Conrady () and the second by means of the thin prism model Brown (). The model employed in the computer vision software library OpenCV OpenCV () integrates a rational term to model radial rotationally invariant distortion with polynomial terms accounting for thin prism and decentering distortion.
Radial rotationally invariant distortion, decentering distortion and thin-prism distortion are examples of models with interesting geometrical properties. They are linear, in the sense that the models constitute a vector space, they are isotropic, i.e., invariant to plane coordinate rotation and, from physical considerations, are formed of functions that are reflection-symmetric with respect to some axis. Some questions arise naturally:
Are decentering and thin-prism distortion the only quadratic models with the three properties mentioned above? Or do they belong to a larger family of models from which we can select a better choice?
How can we combine these models or extend them while keeping all these properties?
Is it necessary to sacrifice some of these properties in order to obtain models with larger number of parameters?
In this work we intend to complement the physical approach to the analysis of lens distortion models with a geometrical perspective. To this purpose we formalize the relevant geometric properties of the models and obtain those that comply with these properties. In this way, we are in conditions to check to what extent the most employed models enjoy these properties and propose extensions that preserve them.
The paper is organized as follows. In section 2 we formalize the concept of lens distortion model and the main geometric properties of interest. In section 3 we study the basic properties of polynomial models introducing their complex representation that will be essential in the later analysis. Section 4 includes the first result of this work, which is the specification of all the possible polynomial linear isotropic lens distortion models. Section 5 elaborates on this result, providing all the models that enjoy the previous properties and at the same time are formed of functions with reflection symmetry. Section 6 analyzes the properties of the most popular polynomial lens distortion models, placing them in the framework introduced by the theoretical results of the previous sections. Some extensions of these models are considered in section 7, that also includes the corresponding experiments. The conclusions are provided in section 8. An appendix at the end gathers the proofs of the theorems.
2 Lens distortion models
2.1 Distortion functions
We will term lens distortion function with distortion center a smooth mapping that keeps fixed and has identity Jacobian at this point. To simplify the formulation we will assume that is at the origin of coordinates. This is not restrictive in most practical situations, since the center of distortion is usually assumed to coincide with the principal point of the projection. Then the distortion function can be written as a mapping of the form
where and . Function will be termed displacement function. With this definition we are separating the linear and non-linear parts of the imaging process, the linear part being associated to the intrinsic parameter matrix. Two interesting analytical properties of lens distortion functions are easy to check:
Each distortion function has a local inverse that is also of the same form.
The composition of two distortions functions is another function of the same form.
Some physical properties of the imaging system have a correspondence with geometric properties of the displacement function. If the lens has perfect rotational symmetry and the image plane is perfectly orthogonal to the lens symmetry axis, the displacement function must be rotationally invariant. Formally, if represents the planar rotation of angle , given by
a displacement function is rotationally invariant if it satisfies
where denotes function composition.
Lack of parallelism between lens and image plane results in an image formation system that is no longer rotationally symmetric, but is symmetric with respect to the plane through the optical axis orthogonal to both lens and image planes. Displacement functions corresponding to this situation should exhibit reflection symmetry with respect to some line through the distortion center (symmetry axis). Formally, if is the reflection leaving invariant the line through the origin with director vector , we have
The displacement function of a lens distortion model can be seen as a vector field on that vanishes at the origin. An orthogonal basis for such vector fields is given by , . Therefore, each displacement function can be written univoquely as the sum of a radial and a tangential displacement functions:
2.2 Distortion models
We define a lens distortion model as a set of set of displacement functions. A model will be termed linear if it is a vector space under the natural operations of sum and multiplication by scalars. Linear models are of practical importance because they greatly simplify the computational processes of obtainment of camera parameters.
A model is isotropic if it is invariant, as a set of functions, with respect to coordinate rotations. It is natural to consider in practice only models having this property because otherwise the characteristics of the model would vary with a rotation of the data. Formally, if is any function of the model , the model is isotropic if there is a such that
We will also pay special attention to those models including only functions that are reflection symmetric with respect to some axis.
3 Polynomial models
3.1 Polynomial lens displacement functions
The th-degree polynomial lens distortion model is the set of displacement functions of the form
where and are polynomials of degree without linear terms, so its Jacobian vanishes. We will also consider homogeneous th-degree polynomial models in which and are homogeneous polynomials of degree .
For an arbitrary degree we define the vector mapping
so that we can express homogeneous displacement functions as
General (i.e., non-homogeneous displacement functions) can be expressed as sum of homogeneous displacement functions, and, consequently, can be represented by sets of matrices.
The simplest case is the quadratic model, corresponding to . In this case the general and the homogeneous cases coincide. The displacement functions are of the form:
that can be expressed in matrix form as
A polynomial radial displacement is of the form
where is a polynomial. As an example we have the well known -coefficient RRI model, given by functions of the form
It is easy to check that all the polynomial radial distortions that are invariant with respect to rotations are of this form.
We define analogously the polynomial tangential displacement functions as those of the form
where is a polynomial.
In the homogeneous case radial displacement functions can be expressed as
and tangential distortion functions as
Therefore radial and tangential displacement functions constitute linear subspaces of dimension of the matrix space , that intersect trivially. Since the dimension of the matrix space is , the functions and in the decomposition (2) are not in general polynomial for a polynomial displacement function. So we have the following proposition.
The sets of th-degree homogeneous radial or tangential displacements constitute isotropic subspaces of dimension of the matrix space , that intersect trivially.
In the quadratic case the radial displacements are those of the form
and the tangential displacements are those of the form
The direct sum of the corresponding linear models is a vector subspace of dimension four of , with which we can identify the set of quadratic distortion functions. Any polynomial distortion function outside this four-dimensional subspace has non-polynomial radial or tangential components.
3.2 Complex polynomial formulation of displacement functions
Polynomial displacement functions (4) can be expressed equivalently as a single complex polynomial in the complex variables and ,
where is any finite set of index pairs such that , , . These polynomials have not been so far, to the authors knowledge, employed to express lens distortion functions, and we will see that they facilitate enormously the geometrical analysis of models.
Conversely, since , we recover from .
In the quadratic case a general complex polynomial is given by
Let us write . The corresponding real polynomial expression will be of the form
If we denote , , , and , , it is easy to check that the correspondence between both sets of parameters is given by
The matrix is invertible as a consequence of the equivalence between both kinds of parameterizations.
Radial and tangential displacement functions are also easily expressed in complex polynomial notation. Since corresponds to the radial vector and to the tangential vector , radial and tangential displacements are given respectively by expressions of the form
where and are real-valued complex polynomials, i.e., such that for any their evaluation is real. It is easy to check that this is equivalent to having coefficients satisfying .
Therefore the complex polynomials that are multiples of represent displacement functions that lie in the space generated by radial and tangential displacement functions. The only monomials that do not lie in this space are those of the form , thus providing a natural complement of that space (see proposition (1)).
4 Linear isotropic models
In this section we aim at obtaining the polynomial models that enjoy at the same time the properties of being linear and rotationally invariant. To this purpose we will make use of the theory of group representations.
4.1 Group representations on polynomial spaces
Given a group , a representation of on a vector space is a group homomorphism
where stands for the group of automorphisms of , i.e., the set of invertible linear mappings .
As an example that will be useful for our purposes, let us consider the group of plane rotations and the vector space of homogeneous polynomials of degree in the variables . The group representation
is simply given by where
where . It is immediate to check that is a linear mapping whose inverse is .
Since is an automorphism of , the elements of the basis of given by the components of (defined in (5)) are transformed into the basis
and so there exists a regular matrix of order such that
For instance, for we have
A vector subspace is called -invariant if for every . A representation is said to be irreducible if there exist no -invariant subspace but the trivial ones, i.e., the null-subspace and itself.
An important property of compact groups as is that any representation is completely reducible, i.e., the associated vector space can be decomposed as being each the restriction of the representation to an irreducible representation Vinberg ().
4.2 Polynomial displacements and geometric transformations
The set of homogeneous displacement functions of degree , is a vector space in which the plane rotation group acts according to equation (3). Specifically, a rotation transforms the mapping into the mapping given by
where and is defined in (1).
Let us consider in more detail the homogeneous case. The displacement function is then given by the equation
where is a matrix. In order to see how matrix in (15) changes with coordinate rotation we substitute in this equation
Thus an homogeneous distortion function transforms itself under the action of a coordinate rotation into another one given by the previous formula. And, in particular, we have that polynomial models, homogeneous or not, are isotropic.
The complex function formulation (13) allows for an easier treatment of coordinate rotation. Using complex numbers, a coordinate rotation of angle can be written as
Let us see how these changes of variables induce a transformation in the complex polynomial. We have
so that the new polynomial is
In the case of monomials, the corresponding transformation is
We will call the number the winding number of the monomial. Table (1) shows a classification of the monomials of degrees from two to five according to their associated winding number.
For degree two a coordinate rotation transform the coefficients according to
4.3 Rotation-invariant distortion functions
We will call invariant monomials those of zero winding number, i.e., those that are invariant with respect to coordinate rotations (18). They are of the form
and therefore there are no invariant monomials of even degree. The displacement functions that do not change under coordinate rotations are those given by complex linear combinations of invariant monomials.
We can write the term corresponding to an invariant monomial as the sum of a radial and a tangential term as
In the case of degree three, the radial and tangential terms correspond respectively to the matrices
The first one corresponds to the cubic (one-parameter) invariant radial distortion of equation (8) and the other one to invariant tangential distortion. Figure 1 shows the action of the corresponding distortion functions on points of a circle and on a grid.
4.4 Linear isotropic models
In this subsection we obtain all the linear isotropic polynomial models of functions of a given maximum degree. In the language of group representations, these are the invariant subspaces of the representation of the planar rotation group on the vector space of displacement functions. As we mentioned in section 4.1, these invariant subspaces are direct sum of irreducible invariant subspaces. Therefore the problem is that of finding these irreducible subspaces.
Some notation will be useful in the sequel. We will denote by the complex vector space of polynomials spanned by the monomials of degree , by the subspace of generated by the monomials with winding number and the subspace generated by all the monomials with winding number , i.e., the non-invariant monomials. Therefore we have
Let us denote by the complex projective line. Its points are equivalence classes
We will denote . Analogously, the real projective line and its points will be denoted as for .
Since and the elements of are kept fixed by the representation, we just have to obtain the irreducible subspaces of . Albeit the set has a natural structure of complex vector space, we are interested in as a real vector space, since we are identifying it with pairs of polynomials in two real variables. We will denote by this real vector space.
The irreducible real subspaces of the representation are the one-dimensional real subspaces of together with the bidimensional subspaces of the form
where , .
Consider the basis of
where we suppose that the monomials are ordered by their winding number . Since
the matrix of the automorphism with respect to is built with diagonal blocks
An irreducible invariant real subspace of must be associated to a pair of complex conjugate eigenvalues, which necessarily are of the form . Therefore must be an irreducible invariant subspace of
Such subspaces are obtained in lemma 1 and are of the form , , , as stated. ∎
Observe that and are the same space if and only if , for some . Otherwise the spaces have trivial intersection.
In degree we have only three monomials, each of them with a different winding number: , () and (). Therefore there are no invariant monomials. Thus a generic polynomial of is of the form , , and a generic polynomial of is of the form . Thus we can parameterize the set of irreducible invariant subspaces by the pair of coefficients , and since, by remark 1, and produce the same space, we have the irreducible subspaces of can be adequately parameterized by the projective points . These subspaces are thus given by
being real-valued, is the space of radial displacements and
is the space of tangential displacements, as takes only pure imaginary values. Since different irreducible subspaces intersect trivially, we have that the direct sum of any two different subspaces of the form (23) is the whole four-dimensional space
In section (6) we will see another interesting decomposition of this space.
In the case of winding number the subspace generated by the only associated monomial,
already coincides with the irreducible invariant subspace .
5 Reflection-symmetric distortion functions
As we have mentioned before, distortion functions that have reflection symmetry with respect to some axis are important in order to model some optical phenomenons. In this section we obtain all the polynomial models that enjoy at the same time the three properties of being linear, isotropic, and being formed by functions with reflection symmetry. We will see that this triple requirement happens to limit severely the dimensionality of the possible models, thus pointing towards the need of relaxing some of the constraints in order to gain flexibility.
5.1 Equations and parameterizations of the variety
The following theorem describes the polynomial displacement functions with reflection symmetry.
A polynomial displacement function
is reflection-symmetric with respect to the axis if and only it satisfies
which is equivalent to have coefficients of the form
and therefore the coefficients satisfy the relation
A reflection with respect to the axis is expressed in terms of complex numbers by the mapping
Therefore a displacement
is reflection-symmetric with respect to this axis if
A straightforward computation shows that this is equivalent to have coefficients satisfying
Writing , with , the equation above implies
From (27), for , denoting , we must have
The equations (26) are sufficient conditions if there exists a monomial with winding number , as it is easy to check. However, in the general case they are not sufficient conditions as the polynomial
In particular, for the invariant monomials () this implies
For degree two the vertically symmetric functions are
and after coordinate rotation we obtain
Let us see that the first two terms can be written as the sum of a radial term and a tangential term. Writing , we have
after coordinate rotation we obtain for these terms
so that in real polynomial form we have
and in real matrix form, including the three terms, we obtain
Figure 2 shows the action of each of these terms on points on a circle and on a grid oriented according to the symmetry axis.
If we consider functions of degree an analogous process leads to the parameterization
where the first term is radial rotationally invariant, the second is radial, the third tangential, and the third is of none of these types. Figure 3 shows the action of each of these terms on points on a circle and on a grid oriented according to the symmetry axis.
5.2 Linear isotropic reflection-symmetric models
The previous results can be employed to obtain a practical description of linear isotropic quadratic models of reflection symmetric functions, given by the following theorem, whose proof is included in the 9.2, in the appendix.
The linear isotropic distortion models with monomials of degree at most constituted by functions with reflection symmetry are those of the form
where the spaces are defined in 4.1, are polynomials with real coefficients, and is a subspace generated by invariant monomials (20) with real coefficients.111Note that if then and that can also be the null vector subspace.
As we saw in example 5, the irreducible subspaces in are the spaces
and the space
and there are not invariant monomials. Therefore the linear isotropic quadratic distortion models constituted by functions with reflection symmetry are the spaces with and . In the first case we have, noting , , and , , we have
Noting , , , , it is easy to check that the real matrix form for these models is
where the first term corresponds to radial distortion and the second to tangential distortion. Therefore the different models of this family are specified by the ration between these two displacement terms.
The functions of the space are those of the form
and with the identification , , have matrix form
Therefore the set of linear isotropic quadratic distortion models with functions with reflection symmetry consists in a one-parameter family (parametrized by the ratio ) and an additional model. All these models are two-dimensional and the ratio of their parameters, determines the symmetry axis according to the relation for the models of the one-parameter family and for the additional model.
Figure 4 provides a topology-preserving representation of the parameter space of the irreducible isotropic linear models of degree two. Each point of the sphere corresponds to a bidimensional isotropic linear model (see equation (23)) within the four-dimensional radial-tangential space. The parameter space is represented as a sphere through the stereographic projection . The blue circle on the sphere corresponds to those of these models that are constituted by functions with reflection symmetry with respect to some axis (i.e., those given by (32)), the red dots on this circle correspond to the radial and tangential models and the green dots correspond to the thin prism and lens decentering models as we will see in the next section. The isolated point corresponds to the space (33), also constituted by functions with reflection symmetry.
6 Application: analysis of some well-known polynomial models
In this section we discuss how the most commonly used lens distortion models fit in the framework presented above.
Decentering distortion Conrady () is an analytical model of the effect of imperfect alignment of the revolution axes of the lens surfaces. The displacement functions of the model are given by the quadratic functions
In our matrix notation, the model is given by the matrices
This model is obviously linear and, as is known from physical considerations, it is isotropic and formed by functions with reflection symmetry. Therefore it must be an instance of the models (32) or (33). It is easy to check that we are in the first case, with coefficients
and taking and in (32).
Thin prism distortion Brown () models the effect of imperfection in the lens manufacturing process and is given by the expression
so that its matrix is
Observe that the displacement is always proportional to . We see again that this is a particular case of (32), now corresponding to the coefficients
and taking and . Therefore these two models correspond to two points in the one-parameter family of models defined by equation (32) as a consequence of theorem 5.1, represented as the green dots in figure 4.
Let us see how these models are combined in practice. The model employed in the Matlab Computer Vision Toolbox Matlab () is the direct sum of three-coefficient RRI distortion (8) and quadratic decentering distortion (34) (named in the documentation “tangential distortion”), i.e., the model is a particular case of (31), given by