Characterizing domains by their limit set

# Characterizing domains by the limit set of their automorphism group

## Abstract.

In this paper we study the automorphism group of smoothly bounded convex domains. We show that such a domain is biholomorphic to a “polynomial ellipsoid” (that is, a domain defined by a weighted homogeneous balanced polynomial) if and only if the limit set of the automorphism group intersects at least two closed complex faces of the set. The proof relies on a detailed study of the geometry of the Kobayashi metric and ideas from the theory of non-positively curved metric spaces. We also obtain a number of other results including the Greene-Krantz conjecture in the case of uniform non-tangential convergence, new results about continuous extensions (of biholomorphisms and complex geodesics), and a new Wolff-Denjoy theorem.

###### Key words and phrases:
Convex domains, Kobayashi metric, holomorphic maps, Wolff-Denjoy theorem, automorphism group

## 1. Introduction

Given a bounded domain let be the automorphism group of , that is the group of biholomorphisms of . The group is a Lie group and acts properly on . When has nice properties there are believed to be few domains with large automorphism group. For instance:

###### Theorem 1.1 (Wong and Rosay Ball Theorem [Ros79, Won77]).

Suppose is a bounded strongly pseudoconvex domain. Then is non-compact if and only if is biholomorphic to the unit ball.

Other bounded domains with large automorphism group and smooth boundary can be constructed using weighted homogeneous balanced polynomials. Given define an associated weight function by

 wtm(α):=d∑i=1αi2mi.

A real polynomial is then called a weighted homogeneous balanced polynomial if there exists some so that

 p(z1,…,zd)=∑wtm(α)=wtm(β)=1/2Cα,βzα¯¯¯zβ.
###### Definition 1.2.

A domain is called a polynomial ellipsoid if

 E={(w,z)∈C×Cd:|w|2+p(z)<1}

where is a weighted homogeneous balanced polynomial.

There are many examples of bounded polynomial ellipsoid, for instance

 {(w,z)∈C×C:|w|2+|z|2m<1}

for any integer . Moreover, a polynomial ellipsoid always has a non-compact automorphism group: when is a weighted homogeneous balanced polynomial the domain

 C:={(w,z)∈C×Cd:Im(w)>p(z)}

has non-compact automorphism group (namely real translations in the first variable and a dilation) and the map given by

 F(z0,…,zd)=(1−iz0/41+iz0/4,z1(1+iz0/4)1/m1,…,zd(1+iz0/4)1/md)

is a biholomorphism of to

 E={(w,z)∈C×Cd:|w|2+p(z)<1}.

In a series of papers Bedford and Pinchuk [BP88, BP91, BP94, BP98] studied the automorphism group of domains of finite type and in particular gave the following characterization of the domains described above:

###### Theorem 1.3.

[BP94] Suppose is a bounded convex domain with boundary and finite type in the sense of D’Angelo. Then is non-compact if and only if is biholomorphic to a polynomial ellipsoid.

There are many other results characterizing special domains via the properties of their automorphism group and boundary, see for instance [GK87, Kim92, Won95, Gau97, Ver09] and the survey paper [IK99]. Like the two theorems mentioned above, almost all previous work assumes that either the entire boundary or a point in the limit set satisfies some infinitesimal condition (for instance strong pseudoconvexity, finite type, or Levi flat). In contrast to these result we provide a new characterization of balanced domains in terms of the geometry of the limit set.

We define the limit set of , denoted , to be the set of points where there exists some and some sequence such that . Since acts properly on , when is non-compact the limit set is non-empty.

If is a bounded convex domain with boundary and let be the complex hyperplane tangent to at . Then the closed complex face of a point is the closed set .

With this language we will prove:

###### Theorem 1.4.

Suppose is a bounded convex domain with boundary. Then the following are equivalent:

1. intersects at least two closed complex faces of ,

2. is biholomorphic to a polynomial ellipsoid.

###### Remark 1.5.

Suppose is a bounded convex domain with boundary and are distinct. Then the following are equivalent:

1. and are in different closed complex faces of ,

2. ,

3. the complex line containing and intersects .

### Acknowledgments

I would like to thank Anders Karlsson for bringing to my attention his work in [Kar05], in particular the general theory of isometries of metric spaces described there motivated some of the results below. I would also like to thank Eric Bedford, Gautam Bharali, and a referee for helpful comments which improved the exposition of this paper. This material is based upon work supported by the National Science Foundation under Grant Number NSF 1400919.

## 2. Outline of the proof of Theorem 1.4

One of the key ideas in the proof of Theorem 1.4 is to show that a smoothly bounded convex domain endowed with its the Kobayashi metric behaves like a Gromov hyperbolic metric space. In this section we will recall some properties of Gromov hyperbolic metric spaces, describe analogues of these properties for the Kobayashi metric on a smoothly bounded convex domain, and then describe the main steps in the proof Theorem 1.4. We then end this section with some other applications of these negative curvature type properties.

### 2.1. Gromov hyperbolic metric spaces

Suppose is a metric space. If is an interval, a curve is a geodesic if for all . A geodesic triangle in a metric space is a choice of three points in and geodesic segments connecting these points. A geodesic triangle is said to be -thin if any point on any of the sides of the triangle is within distance of the other two sides.

###### Definition 2.1.

A proper geodesic metric space is called -hyperbolic if every geodesic triangle is -thin. If is -hyperbolic for some , then is called Gromov hyperbolic.

###### Remark 2.2.

Bridson and Haefliger’s [BH99] book on non-positively curved metric spaces is one of the standard references for Gromov hyperbolic metric spaces.

For a metric space the Gromov product of at is defined to be:

 (p|q)o=12(d(p,o)+d(o,q)−d(p,q)).

If is Gromov hyperbolic, then one can use the Gromov product to define an abstract boundary called the ideal boundary. A sequence is said to be converge to infinity if

 limi,j→∞(pi|pj)o=∞

for some (and hence any) . The set is then the equivalence classes of sequences converging to infinity where two such sequences and are equivalent if

 limi,j→∞(pi|qj)o=∞

for some (and hence any) . Finally, there is a natural topology on which makes it a compactification of (see for instance Chapter III.H Section 3 in [BH99]). It is important to note that when is not Gromov hyperbolic, the relation defined above may not be transitive.

This compactification behaves nicely with respect to 1-Lipschitz maps . In particular, Karlsson proved the following Wolff-Denjoy theorem:

###### Theorem 2.3.

[Kar01, Proposition 5.1] Suppose is Gromov hyperbolic. If is 1-Lipschitz, then either:

1. for every the orbit is bounded in ,

2. there exists some so that

 limn→∞fn(p)=x

for all .

For Gromov hyperbolic metric spaces, it is possible to characterize the isometries in terms of their long term behavior:

###### Definition 2.4.

Suppose is Gromov hyperbolic and is an isometry. Then:

1. is elliptic if the orbit is bounded for some (hence any) ,

2. is hyperbolic if is not elliptic and

 limn→∞φn(p)≠limn→−∞φn(p)

for some (hence any) ,

3. is parabolic if is not elliptic and

 limn→∞φn(p)=limn→−∞φn(p)

for some (hence any) .

###### Remark 2.5.

Notice that Theorem 2.3 implies that every isometry of is either elliptic, hyperbolic, or parabolic.

One more important property of Gromov hyperbolic metric spaces is that geodesics joining two distinct points in the ideal boundary “bend” into the space:

###### Theorem 2.6.

Suppose is Gromov hyperbolic. If and are neighborhoods of in so that , then there exists a compact set with the following property: if is a geodesic with and , then .

###### Remark 2.7.

Conditions of this type were first introduced by Eberlein and O’Neill [EO73] in the context of non-positively curved simply connected Riemannian manifolds. Also see [BGS85, page 54] or [BH99, page 294].

### 2.2. The Kobayashi metric on smoothly bounded convex domains

We now describe how the properties described above extend to the Kobayashi metric on a smoothly bounded convex domain.

Given a domain , let be the Kobayashi distance on . We recently proved the following:

###### Theorem 2.8.

[Zim16] Suppose is a bounded convex domain with boundary. Then the following are equivalent:

1. has finite type in the sense of D’Angelo,

2. is Gromov hyperbolic.

###### Remark 2.9.

Balogh and Bonk [BB00] proved that the Kobayashi metric on a strongly pseudoconvex domain is Gromov hyperbolic.

For convex domains of finite type, the ideal and topological boundary also coincide:

###### Proposition 2.10.

[Zim16, Proposition 11.3, Proposition 11.5] Suppose is a bounded convex domain with finite type in the sense of D’Angelo. If are sequences such that and , then

 limn,m→∞(pn|qm)o=∞ if and % only if x=y.

In particular, the ideal boundary of is homeomorphic to the topological boundary of .

One important step in the proof of Theorem 1.4 is to show that the Gromov product is still reasonably behaved even when the domain does not have finite type:

###### Theorem 2.11.

(see Theorem 4.1 below) Suppose is a bounded convex domain with boundary and are sequences such that and .

1. If , then

 limn,m→∞(pn|qm)o=∞.
2. If

 limsupn,m→∞ (pn|qm)o=∞,

then .

Although this behavior is much weaker than the finite type case, we can still use Theorem 2.11 to prove variants of Theorem 2.3 and Theorem 2.6. For instance, Theorem 2.11 and Karlsson’s [Kar01] work on the behavior of 1-Lipschitz maps on general metric spaces imply the following:

###### Theorem 2.12.

(see Theorem 5.1 below) Suppose is a bounded convex domain with boundary. If is 1-Lipschitz with respect to the Kobayashi metric, then either has a fixed point in or there exists some so that

 limk→∞dEuc(fk(p),TCx∂Ω)=0

for all .

###### Remark 2.13.

Abate and Raissy [AR14] proved Theorem 2.12 with the additional assumption that is .

Using Theorem 2.12 we can characterize the automorphisms of into elliptic, hyperbolic, and parabolic elements. Suppose is a bounded convex domain with boundary and . Then by Theorem 2.12 either has a fixed point in or there exists a complex supporting hyperplane of so that

 limk→∞dEuc(φkp,H+φ)=0

for all . In this latter case, we call the attracting hyperplane of .

###### Definition 2.14.

Suppose is a bounded convex domain with boundary and . Then:

1. is elliptic if has a fixed point in ,

2. is parabolic if has no fixed point in and ,

3. is hyperbolic if has no fixed points in and . In this case we call the repelling hyperplane of .

###### Remark 2.15.

Notice that Theorem 2.12 implies that every automorphism of is either elliptic, hyperbolic, or parabolic.

We can also use Theorem 2.11 to establish a version of Theorem 2.6 for convex domains:

###### Theorem 2.16.

(see Theorem 6.1 below) Suppose is a bounded convex domain with boundary. If and are neighborhoods of , in so that , then there exists a compact set with the following property: if is a geodesic with and , then .

### 2.3. Outline of the proof of Theorem 1.4

The difficult direction of Theorem 1.4 is to show that is biholomorphic to a polynomial ellipsoid when the limit set intersects at least two different closed complex faces. The main steps in the proof of this direction are the following:

1. Use the metric properties described in Subsection 2.2 to show that contains a hyperbolic element and an orbit of this element converges non-tangentially to a boundary point (see Sections 8 and 7).

2. Use a rescaling argument and the metric properties described in Subsection 2.2 to show that has finite type in the sense of D’Angelo (see Section 9).

3. Use another rescaling argument to show that the entire boundary has finite type (see Section 10).

4. Apply Bedford and Pinchuk’s result to deduce that is biholomorphic to a polynomial ellipsoid.

### 2.4. Other applications of Theorem 2.11

In this subsection we describe some other applications of Theorem 2.11 (many of which are used in the proof of Theorem 1.4).

#### Boundary extensions

A convex domain is called -strictly convex if every supporting complex hyperplane intersects at exactly one point. When is this is equivalent to for every . For -strictly convex domains we have the following corollary of Theorem 2.11:

###### Corollary 2.17.

Suppose is a bounded -strictly convex domain with boundary. If are sequences such that and , then

 limn,m→∞(pn|qm)o=∞ if and % only if x=y.

As an application of this corollary we will prove the following result about boundary extensions:

###### Theorem 2.18.

(see Theorem 12.1 below) Suppose and are bounded convex domains with boundaries. If is -strictly convex, then every isometric embedding extends to a continuous map .

In particular:

###### Corollary 2.19.

Suppose are bounded convex domains with boundaries. If is -strictly convex, then every biholomorphism extends to a continuous map .

###### Corollary 2.20.

Suppose is a bounded -strictly convex domain with boundary. Then every complex geodesic extends to a continuous map .

###### Remark 2.21.
1. As mentioned above, Theorem 2.18 is a consequence of the behavior of the Gromov product on convex domains. It is worth mentioning that Forstnerič and Rosay [FR87, Theorem 1.1] also used the behavior of the Gromov product (without using this terminology) to establish continuous boundary extensions of proper holomorphic maps between strongly pseudoconvex domains.

2. There is also an alternative approach to proving boundary extensions of holomorphic maps which uses lower bounds on the infinitesimal Kobayashi metric and a Hardy-Littlewood type lemma, see for instance [DF79, CHL88, Mer93, Bha16]. These arguments only appear to work when the infinitesimal Kobayashi metric obeys some estimate of the form

 kΩ(x;v)≥∥v∥f(dEuc(x,∂Ω))

where , see [BZ16]. It is possible to construct smoothly bounded -strictly convex domains where the infinitesimal Kobayash metric fails to satisfy such estimates.

#### Non-existence of holomorphic maps

We will also use Theorem 4.1 to show that certain holomorphic maps cannot exist:

###### Theorem 2.22.

(see Theorem 9.7 below) Suppose is a bounded convex domain with boundary. Then there does not exist a holomorphic map which induces an isometric embedding .

###### Remark 2.23.
1. If is a Gromov hyperbolic metric space then there does not exist an isometric embedding of into . In particular, the above Corollary shows that convex domains with boundary have some hyperbolic behavior.

2. The statement of Theorem 9.7 below is considerably more general and is used to prove a special case of the Greene-Krantz conjecture (see Theorem 2.25 below).

#### The Greene-Krantz conjecture

The second main step in the proof of Theorem 1.4 is related to an old conjecture of Greene and Krantz. In particular, in the 1990’s Greene and Krantz conjectured:

###### Conjecture 2.24.

[GK93] Suppose that is a bounded pseudoconvex domain with boundary. If , then has finite type in the sense of Kohn/D’Angelo/Catlin.

There are a number of partial results supporting the conjecture, see for instance the survey paper [Kra13]. In Section 9, we will prove the following special case of this conjecture:

###### Theorem 2.25.

(see Theorem 9.1 below) Suppose is a bounded convex domain with boundary. If there exists , , , and so that

 {x+e−2tnx:t>T}⊂Aut(Ω)⋅{p∈Ω:KΩ(p,o)≤M},

then has finite type in the sense of D’Angelo.

Here is the idea of the proof: if had infinite type, then we could use a rescaling argument to construct a holomorphic map having (essentially) the properties in the hypothesis of Theorem 2.22 which is impossible.

## 3. Preliminaries

### 3.1. Notations

1. For let be the standard Euclidean norm and be the standard Euclidean distance.

2. Given an open set , , and let

 δΩ(p)=inf{dEuc(p,x):x∈∂Ω}

and

 δΩ(p;v)=inf{dEuc(p,x):x∈∂Ω∩(p+C⋅v)}.

### 3.2. The Kobayashi metric

Given a domain the (infinitesimal) Kobayashi metric is the pseudo-Finsler metric

 kΩ(x;v)=inf{|ξ|:f∈Hol(Δ,Ω), f(0)=x, d(f)0(ξ)=v}.

By a result of Royden [Roy71, Proposition 3] the Kobayashi metric is an upper semicontinuous function on . In particular if is an absolutely continuous curve (as a map ), then the function

 t∈[a,b]→kΩ(σ(t);σ′(t))

is integrable and we can define the length of to be

 ℓΩ(σ)=∫bakΩ(σ(t);σ′(t))dt.

One can then define the Kobayashi pseudo-distance to be

 KΩ(x,y)=inf{ℓΩ(σ):σ:[a,b]→Ω is absolutely continuous, with σ(a)=x, and σ(b)=y}.

This definition is equivalent to the standard definition by a result of Venturini [Ven89, Theorem 3.1].

A nice introduction to the Kobayashi metric and its properties can be found in [Kob05] or [Aba89].

One important property of the Kobayashi metric on a convex set is the following:

###### Proposition 3.1.

[Bar80] Suppose is a convex domain Then the following are equivalent:

1. is a Cauchy complete geodesic metric space,

2. does not contain any complex affine lines.

### 3.3. The disk and the upper half plane

For the disk and upper half plane the Kobayashi metric coincides with the Poincaré metric.

Let . Then

 kΔ(ζ;v)=|v|1−|ζ|2

and

 KΔ(ζ1,ζ2)=tanh-1∣∣ ∣∣ζ1−ζ21−ζ1¯¯¯¯¯ζ2∣∣ ∣∣.

Next let . Then

 kH(ζ;v)=|v|2Im(ζ)

and

 KH(ζ1,ζ2)=12arcosh(1+|ζ1−ζ2|22Im(ζ1)Im(ζ2)).

### 3.4. Almost geodesics

In the proof of Theorem 1.4 it will often be convenient to work with a class of curves which we call almost-geodesics:

###### Definition 3.2.

Suppose is a metric space and is an interval.

1. A curve is a -quasi-geodesic if

 1A|t−s|−B≤d(σ(s),σ(t))≤A|t−s|+B

for all .

2. If then a curve is an -almost-geodesic if is an -quasi-geodesic and

 d(σ(s),σ(t))≤K|t−s|

for all .

The main motivation for considering almost-geodesics is Proposition 4.3 below which shows that inward pointing normal lines can be parametrized to be an almost-geodesics for convex domains with boundary. We should also note that an -almost-geodesic is a geodesic, thus motivating the choice of additive factor.

## 4. The Gromov product

In this section we prove Theorem 2.11 which we restate:

###### Theorem 4.1.

Suppose is a bounded convex domain with boundary and are sequences such that and .

1. If , then

 limn,m→∞(pn|qm)o=∞.
2. If

 limsupn,m→∞ (pn|qm)o=∞,

then .

We begin by proving a series of lemmas.

###### Lemma 4.2.

Suppose is a convex domain and is a complex hyperplane such that . Then for any we have

 KΩ(z1,z2)≥12∣∣∣logdEuc(H,z1)dEuc(H,z2)∣∣∣.
###### Proof.

Since is convex, there exists a real hyperplane so that and . By translating and rotating , we may assume that

 HR={(z1,…,zd)∈Cd:Im(z1)=0}

and

 Ω⊂{(z1,…,zd)∈Cd:Im(z1)>0}.

Consider the projection given by . Then

 P(Ω)⊂H={w∈C:Im(w)>0}

and so

 KΩ(z1,z2)≥KP(Ω)(P(z1),P(z2))≥KH(P(z1),P(z2)).

Now for we have

 KH(w1,w2) =12arcosh(1+|w1−w2|22Im(w1)Im(w2)) ≥12arcosh(1+(|w1|−|w2|)22|w1||w2|)=12arcosh(|w1|2|w2|+|w2|2|w1|) =12∣∣ ∣∣log(|w1||w2|)∣∣ ∣∣.

So

 KΩ(z1,z2)≥12∣∣∣log|P(z1)||P(z2)|∣∣∣.

Since this implies the lemma. ∎

Suppose is a domain with boundary. If let be the inward pointing normal unit vector at .

###### Proposition 4.3.

Suppose is a bounded convex domain with boundary. Then there exists and so that if then the curve given by

 σx(t)=x+ϵe−2tnx

is an -almost-geodesic.

###### Remark 4.4.

The most difficult inequality to establish in the above proposition is the upper bound

 KΩ(σx(t),σx(s))≤|t−s|+log(K).

To show this we will closely follow the proof of Proposition 2.5 in [FR87].

###### Proof.

For let

 D0:={w∈C:|w|<ρ and C|Im(w)|1+α

For let be the map

 ϕx(w)=x+wnx.

Since is we can pick so that for all .

Next let be a domain with boundary, , and symmetric about the real axis. Such a domain can be obtained by smoothing near the two corner points. Now since is symmetric about the real axis there exists a biholomorphic map with and

 lims→0φ(s)=1.

Since has boundary, extends to a diffeomorphism (see for instance [Gol69, page 426 Theorem 6]).

Now fix and so that

 0≤1−κt≤φ(t)≤1−1κt

for .

Then for we have

 KD0(a,b) ≤KD(a,b)=KΔ(φ(a),φ(b))=12log(1+φ(a))(1−φ(b))(1−φ(a))(1+φ(b)) ≤12log(2)+12log1−φ(b)1−φ(a) ≤12log(2)+log(κ)+12log(b/a).

Thus if we have:

 KΩ(σx(t),σx(s))≤KD0(e−2tϵ,e−2sϵ)≤12log(2)+log(κ)+|t−s|.

On the other hand,

 KΩ(σx(t),σx(s))≥12∣∣∣logdEuc(TCx∂Ω,σx(t))dEuc(TCx∂Ω,σx(s))∣∣∣=|t−s|.

Thus for any the curve is a -quasi-geodesic.

Now since is , by possibly decreasing we can assume that

 {x+wnx:|w|≤2ϵ and 12|Im(w)|≤Re(w)}⊂Ω

for all . This implies that there exists a so that

 δΩ(σx(t);σ′x(t))≥Cϵe−2t

for all and . Then

 Missing or unrecognized delimiter for \left

and so

 KΩ(σx(t),σx(s))≤2C|t−s|.

Thus is a -quasi-geodesic.

Thus for all the curve is an -almost-geodesic with . ∎

###### Lemma 4.5.

Suppose is a bounded convex domain with boundary, , and . Then there exists and such that

 KΩ(p,q)≥12log1δΩ(p)+12log1δΩ(q)−C

when , , and .

###### Remark 4.6.

Abate [Aba89, Proposition 2.4.24, Corollary 2.4.25] proved a weaker version of the above lemma assuming that has boundary and .

###### Proof.

For a set and let

 Nδ(A)={z∈Cd:dEuc(z,A)≤δ}.

For a point let and for a point let

 Π(p):={x∈∂Ω:δΩ(p)=dEuc(p,x)}.

Since is only , we may have for arbitrarily close to .

Next for let

 X(δ):=¯¯¯¯Ω∩(∪{Nδ(TCxp∂Ω):p∈¯¯¯¯Ω∩Nδ(TCx∂Ω) and xp∈Π(p)})

and

 Y(δ):=¯¯¯¯Ω∩(∪{Nδ(TCyq∂Ω):q∈¯¯¯¯Ω∩Nδ(TCy∂Ω) and yq∈Π(q)}).

Notice that and are compact.

We claim that there exists so that . Suppose not, then for each there exists , , , and so that

 ¯¯¯¯Ω∩N1/n(TCxn∂Ω)∩N1/n(TCyn∂Ω)≠∅.

Fix

 zn∈¯¯¯¯Ω∩N1/n(TCxn∂Ω)∩N1/n(TCyn∂Ω)

and pass to a subsequence so that , , and . By construction and . So and . Thus

 z∈¯¯¯¯Ω∩TCx∂Ω∩TCy∂Ω

which contradicts the fact that is convex, is , and . So we can pick so that .

Now since and are compact there exists so that

 Nr(X(δ))∩Nr(Y(δ))=∅.

Now let . Suppose that and . Then and . Moreover, if we pick and then

 Extra open brace or missing close brace

and

 Extra open brace or missing close brace

Now let be a geodesic with and . Since

 Nr(X(δ))∩Nr(Y(δ))=∅

there exists some so that . Then by Lemma 4.2 we have

 KΩ(p,q) =KΩ(p,σ(S))+KΩ(σ(S),q) ≥12logdEuc(σ(S),TCxp∂Ω)dEuc(p,TCxp∂Ω)+12logdEuc(σ(S),TCyq∂Ω)dEuc(q,TCyq∂Ω) ≥12log1δΩ(p)