Metrics, quantization and registration in varifold spaces

# Metrics, quantization and registration in varifold spaces

Hsi-Wei Hsieh Department of Applied Mathematics and Statistics, Johns Hopkins University  and  Nicolas Charon Department of Applied Mathematics and Statistics, Johns Hopkins University
###### Abstract.

This paper is concerned with the theory and applications of varifolds to the representation, approximation and diffeomorphic registration of shapes. One of its purpose is to synthesize and extend several prior works which, so far, have made use of this framework mainly in the context of submanifold comparison and matching. In this work, we instead consider deformation models acting on general varifold spaces, which allows to formulate and tackle diffeomorphic registration problems for a much wider class of geometric objects and lead to a more versatile algorithmic pipeline. We study in detail the construction of kernel metrics on varifold spaces and the resulting topological properties of those metrics, then propose a mathematical model for diffeomorphic registration of varifolds under a specific group action which we formulate in the framework of optimal control theory. A second important part of the paper focuses on the discrete aspects. Specifically, we address the problem of optimal finite approximations (quantization) for those metrics and show a -convergence property for the corresponding registration functionals. Finally, we develop numerical pipelines for quantization and registration before showing a few preliminary results for one and two-dimensional varifolds.

###### Key words and phrases:
varifolds, diffeomorphic registration, reproducing kernels, quantization, optimal control, -convergence

.

## 1. Introduction

Shape is a bewildering notion: while simultaneously intuitive and ubiquitous to many scientific areas from pure mathematics to biomedicine, it remains very challenging to pin down and analyze in a systematic way. The goal of the research field known as shape/pattern analysis is precisely to provide solid mathematical and algorithmic frameworks for tasks such as automatic comparison or statistical analysis in ensembles of shapes, which is key to many applications in computer vision, speech and motion recognition or computational anatomy, among many others.

What makes shape analysis such a difficult and still largely open problem is, on the one hand, the numerous modalities and types of objects that can fall under this generic notion of shape but also the fundamental nonlinearity that is an almost invariable trait to most of the shape spaces encountered in applications. As a result, the seemingly simple issue of defining and computing distances or means on shapes is arguably a research topic of its own, which has generated countless works spanning several decades and involving concepts from various subdisciplines of mathematics. Among many important works, the model of shape space laid out by Grenander in (?, ?) is especially relevant to the present paper. The underlying principle is to build distances between shapes which are induced by metrics on some deformation groups acting on those shapes. This approach has the advantage (at a theoretical level at least) of shifting the problem of metric construction from the many different cases of shape spaces to the single setting of deformation groups. One of the fundamental requirement is the right-invariance of the metrics on those groups; finding the induced distance between two given shapes then reduces to determining a deformation of minimal cost in the group, in other words to solving a registration problem.

Besides usual finite-dimensional groups like rigid of affine transformations, there is in fact a lot of practical interest in applying such an approach with groups of ”large deformations”, specifically groups of diffeomorphisms. This has triggered the exploration of right-invariant metrics over diffeomorphism groups. The Large Deformation Diffeomorphic Metric Mapping (LDDMM) model pioneered in (?, ?, ?) is one of such framework that defines Riemannian metrics for diffeomorphic mappings obtained as flows of time-dependent vector fields (c.f. the brief presentation of Section 4.2). In this setting, registering two shapes can be generically formulated as an optimal control problem, the functionals to optimize being typically a combination of a deformation regularization term given by the LDDMM metric on the group and a fidelity term that enforces (approximate) matching between the two shape objects. Applications of this model have been widespread in particular within the field of computational anatomy, due to the ability to adapt it to various data structures including landmarks, 2D and 3D images, tensor fields… see e.g. (?, ?, ?) for recent reviews.

Interestingly, this line of work has also been drawing many useful concepts from the seemingly distant area of mathematics known as geometric measure theory (?, ?). The key idea of representing shapes (submanifolds) as measures or distributions has been instrumental in the theoretical study of Plateau’s problem on minimal surfaces and more generally in calculus of variations. It can also prove effective for computational purposes, in problems such as discrete curvature approximations (?, ?, ?) or estimation of shape medians (?, ?). With regard to the aforementioned deformation analysis problems, the potential interest of geometric measure theory has been identified early on in the works of (?, ?, ?). Indeed, LDDMM registration of objects like geometric curves or surfaces requires fidelity terms independent of the parametrization of either of the two shapes. On the practical side, this means that one cannot usually rely on predefined pointwise correspondences between the vertices of two triangulated surfaces for instance, which makes the registration problem significantly harder than in the case of labelled objects such as landmarks or images.

The embedding of unparametrized shapes into measure spaces provides one possible way to address the issue, by constructing parametrization-invariant fidelity metrics as restrictions of metrics on those measure spaces themselves. Several competing approaches have been introduced, each relying on embeddings into different spaces of generalized measures: (?, ?, ?, ?) are based on the representation of oriented curves and surfaces as currents, (?, ?) and (?, ?, ?) extended this model to the setting of unoriented and oriented varifolds, while (?, ?, ?) considers the higher-order representation of normal cycles. One common feature to all those works, however, is that they are focused primarily on registration of curves or surfaces. In other words, the use of current, varifolds or normal cycles confines to the computation of a fidelity metric to guide registration algorithms but the deformation model itself remains tied to the curve/surface setting or equivalently, in the discrete situation, to objects described by point set meshes.

The guiding theme and main objective of this paper is to investigate an alternative framework that, in contrast with those prior works, would formulate the deformation model as well as tackle the registration problem directly in these generalized measure spaces: we focus specifically on the (oriented) varifold setting of (?, ?). There are several arguments for the interest of such an approach but in our point of view, the primary motivation lies in the fact that, varifolds being more general than submanifolds, the proposed framework allows to extend large deformation analysis methods to a range of new geometric objects while giving more flexibility to deal with some of the flaws which are commonplace in shapes segmented from raw data. As a proof of concept, our recent work (?, ?) considered the simple case of registration of discrete one-dimensional varifolds. Building on these preliminary results, the present paper intends to provide a thorough and general study of the framework.

The specific contributions and organization of this paper are the following. First, we propose a comprehensive study of the class of kernel metrics on varifold spaces initiated in (?, ?, ?), in particular by examining the required conditions to recover true distances between all varifolds (as opposed to the subset of rectifiable varifolds) and comparing the resulting topologies with some standard metrics on measures. This is presented in Section 3 after the brief introduction to the notion of oriented varifold of Section 2. In Section 4, we discuss the action of diffeomorphisms and from there derive a formulation of LDDMM registration of general varifolds, for which we show the existence of solutions and derive the Hamiltonian equations associated to the corresponding optimal control problem. Section 5 addresses the issue of quantization in varifold space, namely of approximating any varifold as a finite sum of Dirac masses. We consider a novel approach in this context, that consists in computing projections onto particular cones of discrete varifolds. We then prove the -convergence of the corresponding approximate registration functionals. In Section 6, we derive the discrete version of the optimal control problem and optimality equations, from which we deduce a geodesic shooting algorithm for the diffeomorphic registration of discrete varifolds. Finally, results on - and - varifolds are presented in Section 7, emphasizing the potentiality of the approach to tackle data structures which are typically challenging for previous algorithms that are designed for point sets and meshes.

## 2. The space of oriented varifolds

The concept of varifold was originally developed in the context of geometric measure theory by (?, ?), (?, ?) and (?, ?) for the study of Plateau’s problem on minimal surfaces. The interest in registration and shape analysis was evidenced in (?, ?, ?). In those works, varifolds provide a convenient representation of geometric shapes such as rectifiable curves and surfaces and an efficient approach to define and compute fidelity terms for registration, or to perform clustering, classification in those shape spaces. The main purpose of this section is to introduce varifolds in this latter context. The case of non-oriented shapes was thoroughly investigated in (?, ?). Later on, the generalized framework of oriented varifold was proposed in (?, ?) but only for objects of dimension or co-dimension one. In the following, we provide a fully general presentation of oriented varifolds and their properties, that also does not specifically focus on the case of rectifiable varifolds as these previous works did. Although we assume here that all the considered shapes are oriented, we emphasize that the non-oriented framework of (?, ?) can be recovered almost straightforwardly through adequate choices of orientation-invariant kernels as we shall briefly point out later on.

### 2.1. Definition

The underlying principle of varifolds is to extend measures of by incorporating an additional tangent space component. In this work, we will consider such spaces to be oriented. Thus, for a given dimension , we first need to introduce the set of all possible -dimensional oriented tangent spaces in :

###### Definition 2.1.

The -dimensional oriented Grassmannian is the set of all oriented -dimensional linear subspaces of .

The oriented Grassmannian is a compact manifold of dimension which can be identified to the quotient . It is also a double cover of the (non-oriented) Grassmannian of -dimensional subspaces of . For practical purposes, a more convenient representation of is the one detailed in the following remark.

###### Remark 2.2.

Given , there exists a basis of such that has consistent orientation with . Then the following map, called the oriented Plücker embedding, is well defined and injective,

 iP: ˜Gnd ↦{ξ∈Λd(Rn):|ξ|=1} T ↦u1∧⋯∧ud|u1∧⋯∧ud|.

This allows to identify as a subset of the unit sphere of which inherits the topology of the inner product on . We remind that this inner product is defined for any , in by the determinant of the Gram matrix:

 (1) ⟨ξ,η⟩=det(ξi⋅ηj)i,j=1,…,d

Through this identification, one can also define the action of linear transformations on as follows

 (2) A⋅T:=Au1∧⋯∧Aud|Au1∧⋯∧Aud|

for any and a linear invertible map.

Similar to the definition of classical varifolds in (?, ?), we define oriented varifolds as measures on .

###### Definition 2.3.

An oriented -varifold on is a nonnegative finite Radon measure on the space . Its weight measure is defined by for all Borel subset of . We denote by the space of all oriented -varifolds.

In the rest of the paper, with a slight abuse of vocabulary, we will often use the word varifold instead of oriented varifold for the sake of concision. Recall that from the Riesz representation theorem, we can alternatively view any varifold as a distribution, i.e. an element of the dual space , where denotes the set of continuous functions vanishing at infinity on . It is defined for any test function by:

 (3) (μ|ω)≐∫Rn×˜Gndω(x,T)dμ(x,T).

As an additional note, another useful representation of a general varifold in can be obtained by the disintegration theorem (see (?, ?) Chap. 2). Namely, if , for -almost every in , there exists a probability measure on such that is -measurable and we can write

 (4) (μ|ω)=∫Rn∫˜Gndω(x,T)dνx(T)d|μ|(x).

In other words, the varifold can be decomposed as its weight measure on together with a family of tangent space probability measures on the Grassmannian at the different points in the support of . This is usually referred to as the Young measure representation of .

### 2.2. Diracs and rectifiable varifolds

There are a few important families of varifolds which will be relevant for the following. First of those are the Diracs. For and , the associated Dirac varifold acts on functions of by the relation

 (δ(x,T)|ω)=ω(x,T), ∀ω∈C0(Rn×˜Gnd).

can be viewed as a singular particle at position that carries the oriented -plane .

A second particular class is the one of rectifiable varifolds, which are in essence the varifolds representing an oriented shape of dimension . More precisely, given an oriented -dimensional submanifold of of finite total -volume, denoting by the oriented tangent space at , one can associate to the varifold , which is defined for all Borel subset by . Here, is the -dimensional Hausdorff measure on , i.e. the measure of -volume of subsets of (we refer the reader to (?, ?) for the precise construction and properties of Hausdorff measures). It is then not hard to see that, as an element of ,

 (μX|ω) =∫Rd×˜Gndω(x,T)dμX(x,T) (5) =∫Xω(x,TX(x))dHd(x).

Such a representation can be extended to slightly more general objects known as oriented rectifiable sets. A subset of is said to be a countably -rectifiable set if , where are Lipschitz function for all (c.f. (?, ?)). We say that is an oriented rectifiable set if is a countably -rectifiable set and is a -measurable function such that for , is the approximate tangent space of at with specified orientation. Rectifiable subsets include both usual submanifolds but also piecewise smooth objects like polyhedra. Given any oriented rectifiable set , we can associate a varifold that we also write given again by (2.2). The set of those will be referred to as the rectifiable oriented varifolds in this paper (note that this is actually more restrictive than the standard definition of rectifiable varifold in the literature which also incorporates an additional multiplicity function).

###### Remark 2.4.

Rectifiable varifolds still make a very ”small” subset of : indeed, in the Young measure representation of (4), we have in this case the very particular constraint that probability measures are Dirac masses, specifically .

## 3. Metrics on varifolds

In this section, we address the issue of defining adequate metrics on the space . After reviewing some classical metrics and their limitations for the specific applications of this work, we turn to metrics defined through positive definite kernels, for which we extend previous constructions introduced in e.g. (?, ?, ?) and derive the most relevant properties of this class of distances.

### 3.1. Standard topologies and metrics on Vd

As a measure/distribution space, can be equipped with various topologies and metrics, several of which have been regularly used in various contexts. We discuss a few of those below.

• mass norm: with the previous identification of measures in with elements of the dual , one can define the following dual metric on :

 (6) dop(μ,ν)≐sup|ω|∞≤1(μ−ν|ω), ∀μ∈Vd.

where . This metric is generally too strong for applications in shape analysis and leads to a discontinuous behavior. Indeed, one can easily verify that for any two Dirac masses and , whenever .

• weak-* topology: a sequence of -varifolds converges to in the weak-* topology (denoted by ) if and only if for all (continuous compactly supported function)

 (7) limi→∞(μi|ω)=(μ|ω).

In fact, the weak-* topology on can be metrized by the following distance:

 d∗(μ,ν)=∑k∈N2−k|(μ−ν|ωk)|,

where is a dense sequence in .

• Wasserstein metric: the Wasserstein-1 distance of optimal transport can be expressed in its Kantorovitch dual formulation (?, ?) as

 (8) dWass1(μ,ν)≐supLip(ω)≤1|(μ−ν|ω)|.

where the sup is taken over all Lipschitz regular functions on with Lipschitz constant smaller than one. This metric is however well-suited for measures with the same total mass. Several recent works (?, ?, ?) have instead proposed generalized Wasserstein distances derived from unbalanced optimal transport.

• Bounded Lipschitz metric: similar to the previous, the bounded Lipschitz distance (sometimes referred to as the flat metric) on is defined by

 (9) dBL(μ,ν)≐sup∥ω∥∞,Lip(ω)≤1|(μ−ν|ω)|.

It can be shown (cf. Ch 8 in (?, ?)) that metrizes the narrow topology on , namely the topology for which a sequence converges to if and only if for all bounded continuous functions .

Clearly, the narrow topology is stronger than the weak-* topology. Furthermore, it is also well known that locally metrizes the weak-* topology on , namely:

###### Proposition 3.1.

Let and be varifolds such that the sequence is tight. Then if and only if .

###### Proof.

Since metrizes the narrow topology, it suffices to show that converges to in the narrow topology. Let be a bounded continuous function defined on and . By the tightness property, we may choose a compact set such that . Let be an open ball that contains . Define

 η(x,T)≐{ω(x,T), if (x,T)∈K0, if (x,T)∈Bc

From Tietz extension theorem, there exists a continuous extension of on such that and . This implies that

 ∣∣∣∫Rn×˜Gndωd(μi−μ)∣∣∣ ≤∫Rn×˜Gnd|ω−~ω|d(μi+μ)+∣∣∣∫Rn×˜Gnd~ωd(μi−μ)∣∣∣ ≤∣∣∣∫Rn×˜Gnd~ωd(μi−μ)∣∣∣+ε.

Taking on both sides, we see that

 limsupi→∞∣∣∣∫Rn×˜Gndωd(μi−μ)∣∣∣<ε.

Since is arbitrary, we obtain that converges to in the narrow topology. ∎

As a direct consequence of Proposition 3.1, we have in particular that weak-* convergence and convergence in are equivalent if one restricts to varifolds that are supported in a fixed compact subset of . Note also that a very similar result to Proposition 3.1 holds when replacing the bounded Lipschitz distance by generalized Wasserstein metrics, as proved in (?, ?).

The above metrics on varifolds all originate from classical ones in standard measure theory. Unlike the mass norm, Wasserstein and bounded Lipschitz metrics have nice theoretical properties in terms of shape comparison. However, for the purpose of diffeomorphic registration that we shall tackle below, one needs metrics that are easy to evaluate numerically. This is typically not the case of and expressed above as there is no straightforward way to compute the corresponding suprema over the respective sets of test functions. One line of work has been considering approximations of optimal transport distances with e.g. entropic regularizers for which Sinkhorn-based algorithms can be derived, see for instance the recent work (?, ?). In this paper, we focus on the alternative approach previously developed for currents in (?, ?) and unoriented varifolds in (?, ?) which instead relies on particular Hilbert spaces of test functions, as we detail in the next section.

### 3.2. Kernel metrics

In this section, we start by defining a general class of pseudo-metrics on based on positive definite kernels and their corresponding reproducing kernel Hilbert space (RKHS). We will then study sufficient conditions on such kernels to recover true metrics before examining the relationship between those kernel metrics and the ones of Section 3.1.

#### 3.2.1. Kernels for varifolds

We refer the reader to (?, ?, ?, ?) for a presentation of the construction and main properties of positive kernels and Reproducing Kernel Hilbert Spaces which we do not recall in detail here for the sake of concision. In the context of varifolds, we are interested in defining positive definite kernels on the product . Along the lines of previous works like (?, ?, ?), we restrict to separable kernels for which we have:

###### Proposition 3.2.

Let and be continuous positive definite kernels on and respectively. Assume in addition that for any , . Then is a positive definite kernel on and the RKHS associated to is continuously embedded in i.e. there exists such that for any , we have .

We recall that the tensor product kernel has the exact expression . The proof of Proposition 3.2 is a straightforward adaptation of the same result for unoriented varifolds (cf. (?, ?) Proposition 4.1).

###### Remark 3.3.

To simplify the rest of the presentation and in the perspective of later numerical considerations, we will also assume specific forms for and , namely that is a translation/rotation invariant radial kernel , with , and where is the inner product on inherited from introduced in remark 2.2. These assumptions are quite natural as they will eventually induce metrics on varifolds invariant to the action of rigid motion, as we shall explain later. Note that the unoriented framework of (?, ?) can be also recovered in this setting by simply restricting to orientation-invariant kernels i.e. such that for all .

Now, if we let be the continuous embedding given by Proposition 3.2 and its adjoint, for any , we have

 (10) (ι∗μ|ω)=∫Rd×˜Gndω(x,T)dμ(x,T), ∀ω∈W.

With (10), we may identify as an element of the dual RKHS . Note that is not injective in general, in other words one can have in but in .

In any case, one can compare any two varifolds through the Hilbert norm of by defining:

 (11) dW∗(μ,μ′)2=∥μ−μ′∥2W∗=∥μ∥2W∗−2⟨μ,μ′⟩W∗+∥μ′∥2W∗

where we use the small abuse of notation of writing and instead of and on the two right hand sides. Due to the potential non-injectivity of , in general only induces a pseudo-metric on .

The main advantage of this construction is that can be expressed more explicitly based on the reproducing kernel property of . Indeed, given any and in , the inner product between them is given by

 ⟨μ,μ′⟩W∗ =∫(Rd×˜Gnd)2kpos(x,x′)kG(T,T′)dμ(x,T)dμ′(x′,T′) (12) =∫(Rd×˜Gnd)2ρ(|x−x′|2)γ(⟨T,T′⟩)dμ(x,T)dμ′(x′,T′)

for kernels selected as in Remark 3.3.

#### 3.2.2. Characterization of distances

As mentioned above, is a priori a pseudo-distance between varifolds. It’s a natural question to ask under which conditions it leads to an actual distance.

Most past works have addressed this question focusing on the case of varifolds representing submanifolds and reunion of submanifolds (?, ?, ?). We can first provide an extension of these results to the general case of oriented rectifiable varifolds. A key notion for the rest of this section is the one of -universality of kernels:

###### Definition 3.4.

A positive definite kernel on a metric space is called -universal when its RKHS is dense in for the uniform convergence topology.

-universality has been studied in great length in such works as (?, ?, ?). In particular, one can provide characterizations of -universality for certain classes of kernels and spaces . In the case of translation-invariant kernels on for instance, it has been established that -universal kernels are the ones which can be expressed through the Fourier transform of finite Borel measures with full support on , which includes: compactly-supported kernels, Gaussian kernels, Laplacian kernels… With the previous definition, we have the following sufficient condition:

###### Theorem 3.5.

Suppose is a -universal kernel on , and . Let and be two oriented -rectifiable sets with , . If , then and -.

The full proof can be found in the Appendix. Note that the first part of the proof directly gives an equivalent statement for unoriented rectifiable varifolds (if one instead assumes for all ), generalizing the result of (?, ?).

However, the previous proposition does not necessarily lead to a distance on the full space . Counter-examples in the case are discussed for example in (?, ?). To recover a true distance on , one needs the previous map or equivalently the map

 (13) μ↦∫Rd×˜Gndk(⋅,(y,T))dμ(y,T), μ∈C0(Rd×˜Gnd)∗

to be injective. As follows from Theorem 6 in (?, ?), this is in fact guaranteed when the kernel on the product space is -universal, specifically

###### Theorem 3.6.

The pseudo-distance induces a distance between signed measures of if and only if is -universal on . In particular, a sufficient condition for to be a distance on is that and are -universal kernels on and respectively.

Note that these conditions are more restrictive than in Theorem 3.5. To our knowledge, there is no simple characterization for general -universal kernels on the Grassmannian. However, within the setting of Remark 3.3, one easily constructs -universal kernels by restriction (based on the Plücker embedding) of -universal kernels defined on the vector space .

#### 3.2.3. Comparison with classical metrics

We now study more precisely the topology induced by the (pseudo) distance on in comparison with the ones defined in Section 3.1. First of all, we observe that, for any with , one must have , where is the embedding constant of Proposition 3.2. Thus, for any and in , we have

 (14) ∥μ−μ′∥W∗=supω∈W, ∥ω∥W≤1∫Rd×˜Gndω d(μ−μ′)≤cWdop(μ,μ′).

From the above inequalities we see that convergence in implies convergence in .

###### Remark 3.7.

With more assumptions on the regularity of the kernel , namely if is continuously embedded in , following a similar reasoning as above, one obtains the bound .

Suppose converges to in narrow topology. Since the map is continuous with respect to the narrow topology, we have

 ∥μi∥2W∗ =∫(Rd×˜Gnd)2k((x,S),(y,T))dμi(x,S)dμi(y,T) →∫(Rd×˜Gnd)2k((x,S),(y,T))dμ(x,S)dμ(y,T) =∥μ∥2W∗,

as . Also, it’s clear that and hence with respect to . To summarize the discussion above:

###### Proposition 3.8.

Let and be varifolds in and assume that with respect to the operator norm or the narrow topology, then in .

###### Remark 3.9.

We emphasize that the result of Proposition 3.8 only requires the assumptions of Proposition 3.2 and thus holds whether is injective or not.

As for the weak-* topology, with the -universality assumption of Theorem 3.6 and restricting to varifolds with bounded total mass, we show that induces a topology stronger than weak-* convergence:

###### Proposition 3.10.

If is -universal, then the topology induced by is finer than the weak-* topology on for any fixed .

###### Proof.

Let and be varifolds in and assume that . For any and , there exists a such that . Then we obtain that from the following inequalities:

 |(μi−μ|f)|≤|(μi|f−g)|+|(μ|g−f)|+|(μi−μ|g)|≤ε+∥μi−μ∥W∗∥g∥W.

Note that the topology induced by may be strictly finer on . Indeed, if , consider , where and fixed. Then while for all . Yet, by combining Propositions 3.1, 3.8 and 3.10, we have the following

###### Corollary 3.11.

Let and be a compact subset. If is -universal, then metrizes the weak-* convergence of varifolds on

In summary, -universality provides a sufficient condition to obtain actual distances between varifolds that can be expressed based on the kernel function. Furthermore, the resulting topology is locally equivalent to the weak-* topology as well as the topology induced by the bounded Lipschitz distance. This equivalence will be of importance in Section 5.

## 4. Deformation and registration of varifolds

Having defined a way of comparing general varifolds through the above kernel metrics , our goal is now to focus on deformation models for those objects in order to formulate and study the diffeomorphic registration problem on .

### 4.1. Deformation models

In this section, we discuss different models for how varifolds can be transported by a diffeomorphism of , in other words what are possible group actions of the diffeomorphism group on .

Let us start by considering the case of an oriented rectifiable subset . A diffeomorphism transports as

 ϕ⋅(X,TX)≐(ϕ(X),Tϕ(X)),

where the transported orientation map writes

 Tϕ(X)(y)≐dϕ−1(y)ϕ⋅TX(ϕ−1(y))

the above term being well-defined from (2). This suggests introducing the following pushforward action on , which is defined for all and by:

 (15) (ϕ#μ|ω)≐∫Rd×˜Gndω(ϕ(x),dxϕ⋅T)JTϕ(x)dμ(x,T)

in which denotes the determinant of the Jacobian of along (i.e. the change of d-volume induced by along at ) which is given by

 JTϕ(x)=det((dxϕ(ei)⋅dxϕ(ej))i,j=1,…,d)

for an orthonormal basis of . One easily verifies that defines a group action which commutes with the action on oriented rectifiable sets, namely

###### Proposition 4.1.

For any oriented rectifiable set and diffeomorphism .

This follows from the area formula for integrals over rectifiable sets, c.f. (?, ?) Chapter 2.

###### Remark 4.2.

This pushforward action also extends the diffeomorphic transport of measures with densities on . Indeed if with a measurable density function on and the Lebesgue measure, we can extend to a n-varifold in by taking a constant global orientation in . Then, for any orientation-preserving diffeomorphism , (15) writes in this case: with is the full Jacobian determinant of , leading to the usual action on densities .

However, in contrast with past works on submanifold registration, this is not the only possible group action that could be considered on the space . For instance, one can define another action by removing the above volume change term, taking instead

 (ϕ∗μ|ω):=∫Rd×˜Gndω(ϕ(x),dxϕ⋅T)dμ(x,T).

This normalized action has the property of preserving the total mass of the varifold, i.e.,

 |ϕ∗μ|(Rn)=|μ|(Rn), ∀μ∈Vd and ϕ∈Diff(Rn).

Although this action is not consistent with the action on rectifiable sets as in Proposition 4.1, this model may be more adequate in applications to certain types of data in which mass preservation is natural.

We refer the interested reader to (?, ?) for a more in depth discussion on the properties (orbits, isotropy subgroups…) of these group actions in the simpler case of 1-varifolds. In the rest of the paper, we will restrict ourselves to the pushforward action model of (15), although we expect the following derivations to adapt to other cases as well, which precise study is for now left as future work.

### 4.2. The diffeomorphic registration problem

With the group action defined above, we are now ready to introduce the mathematical formulation of the diffeomorphic registration problem for general varifolds in . As deformation model, we will rely on the Large Deformation Diffeomorphic Metric Mapping (LDDMM) setting mentioned in the introduction.

Let us briefly sum up the basic construction of LDDMM, which details can be found in (?, ?, ?). In this framework, deformations consist of diffeomorphisms generated by flowing time-dependent vector fields. Let be a fixed RKHS of vector fields on and be space of time dependent velocity fields such that for all , belongs to . The flow map is defined for all by and the ODE . If is continuously embedded in , one can show that for all , is a -diffeomorphism of . Moreover, on the subgroup of , one can define the following right-invariant Riemannian metric:

 dGV(id,ϕ)=inf{∫10∥vt∥2Vdt | φv1=ϕ}

Let us now consider a source (or template) varifold as well as a target . With the above deformation model and metric, registering to consists in finding a deformation that minimizes with the constraint that is close to in the sense of a kernel metric defined in Section 3.2. This can be reformulated as the following optimal control problem:

 (16) argminv∈L2([0,1],V){E(v)=12∫10∥vt∥2Vdt+λ∥μ(1)−μtar∥2W∗}

with being the control, the total cost and the state equation is given by for the pushforward model. The first term in (16) is the regularization term that constrains the regularity of the estimated deformation paths. The second term measures the similarity between the deformed varifold and the target varifold . is a weight parameter between the regularization and fidelity terms. Note that this is consistent with the generic inexact registration problem formulation in LDDMM that was proposed for objects like images, landmarks, submanifolds…

The well-posedness of the optimal control problem (16) holds under the following assumptions:

###### Theorem 4.3.

If is continuously embedded in , is continuously embedded in and , for some compact subset of , then there exists a global minimizer to the problem (16).

The proof is similar to previous results of the same type on rectifiable currents and varifolds. We give it in Appendix for the sake of completeness.

###### Remark 4.4.

One can derive necessary and sufficient conditions on the kernels of and for the two embedding assumptions of Theorem 4.3 to hold (see for instance Theorem 2.11 in (?, ?)). In our context, in order to get for instance, it is enough to assume that and are functions such that all derivatives of up to order 2 vanish as .

As an important note, the formulation of (16) extends registration of submanifolds or rectifiable subsets in the sense that if and for two oriented d-rectifiable subsets of then (16) becomes equivalent, thanks to Proposition 4.1, to registering rectifiable subsets, i.e. to the problem

 argminv∈L2([0,1],V){12∫10∥vt∥2Vdt+λ∥μX(1)−μXtar∥2W∗}

with , which is the setting of many past works as for instance (?, ?, ?, ?).

### 4.3. General optimality conditions

A last important question we address in this section is the derivation of necessary optimality conditions for the solutions of (16). In standard finite-dimensional optimal control problems, these are provided by the Pontryagin Maximum Principle (PMP) introduced originally in (?, ?). The approach generalizes, with a certain number of technicalities, to a broad class of infinite-dimensional shape matching problems, as developed in (?, ?).

We follow the same setting as well as related works such as (?, ?) by first rewriting the above problem as an optimal control problem on diffeomorphisms, i.e.

 argminv∈L2([0,1],V){12∫10∥vt∥2Vdt+g(φv1) | s.t. ˙φvt=vt∘φvt}

with . The state variables are now given by the deformations which we view as elements of the Banach space . Let us denote, for , the mapping . We then introduce the Hamiltonian functional defined by:

 (17) H(p,ϕ,v)=(p|v∘ϕ)−12∥v∥2V

where is the costate variable which is a vector distribution of and denotes the duality bracket in . With the assumptions of Theorem 4.3, it follows from the maximum principle shown in (?, ?) that if is a global minimum of the optimal control problem, there exists a path of costates such that the following equations hold:

 (18) ⎧⎪⎨⎪⎩˙φvt=∂pH(pt,φvt,vt)˙pt=−∂ϕH(pt,φvt,vt)∂vH(pt,ϕvt,vt)=0

with the end time boundary conditions . From the last equation in (18), we can attempt to deduce the form of the optimal . Introducing the Riesz isometry operator and its inverse , we get:

 (19) ξ∗φvtpt−LVvt=0 ⇒ vt=KVξ∗φvtpt.

One additional consequence of (18) is the following conservation of momentum again proved in (?, ?): for all and ,

 (20) (pt|dφvtu)=(p0|u).

Note that (18), (19) and (20) are generic to the LDDMM model and so far independent of the nature of the deformed objects and of the term in the cost. This dependency is entirely encompassed by the boundary condition which we may describe a little more precisely based on the following:

###### Proposition 4.5.

The end-time momentum is a vector distribution in of the form

 (p1|u)= ∫Rn<