Characterizing domains by the limit set of their automorphism group
Abstract.
In this paper we study the automorphism group of smoothly bounded convex domains. We show that such a domain is biholomorphic to a “polynomial ellipsoid” (that is, a domain defined by a weighted homogeneous balanced polynomial) if and only if the limit set of the automorphism group intersects at least two closed complex faces of the set. The proof relies on a detailed study of the geometry of the Kobayashi metric and ideas from the theory of nonpositively curved metric spaces. We also obtain a number of other results including the GreeneKrantz conjecture in the case of uniform nontangential convergence, new results about continuous extensions (of biholomorphisms and complex geodesics), and a new WolffDenjoy theorem.
Key words and phrases:
Convex domains, Kobayashi metric, holomorphic maps, WolffDenjoy theorem, automorphism group2010 Mathematics Subject Classification:
1. Introduction
Given a bounded domain let be the automorphism group of , that is the group of biholomorphisms of . The group is a Lie group and acts properly on . When has nice properties there are believed to be few domains with large automorphism group. For instance:
Theorem 1.1 (Wong and Rosay Ball Theorem [Ros79, Won77]).
Suppose is a bounded strongly pseudoconvex domain. Then is noncompact if and only if is biholomorphic to the unit ball.
Other bounded domains with large automorphism group and smooth boundary can be constructed using weighted homogeneous balanced polynomials. Given define an associated weight function by
A real polynomial is then called a weighted homogeneous balanced polynomial if there exists some so that
Definition 1.2.
A domain is called a polynomial ellipsoid if
where is a weighted homogeneous balanced polynomial.
There are many examples of bounded polynomial ellipsoid, for instance
for any integer . Moreover, a polynomial ellipsoid always has a noncompact automorphism group: when is a weighted homogeneous balanced polynomial the domain
has noncompact automorphism group (namely real translations in the first variable and a dilation) and the map given by
is a biholomorphism of to
In a series of papers Bedford and Pinchuk [BP88, BP91, BP94, BP98] studied the automorphism group of domains of finite type and in particular gave the following characterization of the domains described above:
Theorem 1.3.
[BP94] Suppose is a bounded convex domain with boundary and finite type in the sense of D’Angelo. Then is noncompact if and only if is biholomorphic to a polynomial ellipsoid.
There are many other results characterizing special domains via the properties of their automorphism group and boundary, see for instance [GK87, Kim92, Won95, Gau97, Ver09] and the survey paper [IK99]. Like the two theorems mentioned above, almost all previous work assumes that either the entire boundary or a point in the limit set satisfies some infinitesimal condition (for instance strong pseudoconvexity, finite type, or Levi flat). In contrast to these result we provide a new characterization of balanced domains in terms of the geometry of the limit set.
We define the limit set of , denoted , to be the set of points where there exists some and some sequence such that . Since acts properly on , when is noncompact the limit set is nonempty.
If is a bounded convex domain with boundary and let be the complex hyperplane tangent to at . Then the closed complex face of a point is the closed set .
With this language we will prove:
Theorem 1.4.
Suppose is a bounded convex domain with boundary. Then the following are equivalent:

intersects at least two closed complex faces of ,

is biholomorphic to a polynomial ellipsoid.
Remark 1.5.
Suppose is a bounded convex domain with boundary and are distinct. Then the following are equivalent:

and are in different closed complex faces of ,

,

the complex line containing and intersects .
Acknowledgments
I would like to thank Anders Karlsson for bringing to my attention his work in [Kar05], in particular the general theory of isometries of metric spaces described there motivated some of the results below. I would also like to thank Eric Bedford, Gautam Bharali, and a referee for helpful comments which improved the exposition of this paper. This material is based upon work supported by the National Science Foundation under Grant Number NSF 1400919.
2. Outline of the proof of Theorem 1.4
One of the key ideas in the proof of Theorem 1.4 is to show that a smoothly bounded convex domain endowed with its the Kobayashi metric behaves like a Gromov hyperbolic metric space. In this section we will recall some properties of Gromov hyperbolic metric spaces, describe analogues of these properties for the Kobayashi metric on a smoothly bounded convex domain, and then describe the main steps in the proof Theorem 1.4. We then end this section with some other applications of these negative curvature type properties.
2.1. Gromov hyperbolic metric spaces
Suppose is a metric space. If is an interval, a curve is a geodesic if for all . A geodesic triangle in a metric space is a choice of three points in and geodesic segments connecting these points. A geodesic triangle is said to be thin if any point on any of the sides of the triangle is within distance of the other two sides.
Definition 2.1.
A proper geodesic metric space is called hyperbolic if every geodesic triangle is thin. If is hyperbolic for some , then is called Gromov hyperbolic.
Remark 2.2.
Bridson and Haefliger’s [BH99] book on nonpositively curved metric spaces is one of the standard references for Gromov hyperbolic metric spaces.
For a metric space the Gromov product of at is defined to be:
If is Gromov hyperbolic, then one can use the Gromov product to define an abstract boundary called the ideal boundary. A sequence is said to be converge to infinity if
for some (and hence any) . The set is then the equivalence classes of sequences converging to infinity where two such sequences and are equivalent if
for some (and hence any) . Finally, there is a natural topology on which makes it a compactification of (see for instance Chapter III.H Section 3 in [BH99]). It is important to note that when is not Gromov hyperbolic, the relation defined above may not be transitive.
This compactification behaves nicely with respect to 1Lipschitz maps . In particular, Karlsson proved the following WolffDenjoy theorem:
Theorem 2.3.
[Kar01, Proposition 5.1] Suppose is Gromov hyperbolic. If is 1Lipschitz, then either:

for every the orbit is bounded in ,

there exists some so that
for all .
For Gromov hyperbolic metric spaces, it is possible to characterize the isometries in terms of their long term behavior:
Definition 2.4.
Suppose is Gromov hyperbolic and is an isometry. Then:

is elliptic if the orbit is bounded for some (hence any) ,

is hyperbolic if is not elliptic and
for some (hence any) ,

is parabolic if is not elliptic and
for some (hence any) .
Remark 2.5.
Notice that Theorem 2.3 implies that every isometry of is either elliptic, hyperbolic, or parabolic.
One more important property of Gromov hyperbolic metric spaces is that geodesics joining two distinct points in the ideal boundary “bend” into the space:
Theorem 2.6.
Suppose is Gromov hyperbolic. If and are neighborhoods of in so that , then there exists a compact set with the following property: if is a geodesic with and , then .
2.2. The Kobayashi metric on smoothly bounded convex domains
We now describe how the properties described above extend to the Kobayashi metric on a smoothly bounded convex domain.
Given a domain , let be the Kobayashi distance on . We recently proved the following:
Theorem 2.8.
[Zim16] Suppose is a bounded convex domain with boundary. Then the following are equivalent:

has finite type in the sense of D’Angelo,

is Gromov hyperbolic.
Remark 2.9.
Balogh and Bonk [BB00] proved that the Kobayashi metric on a strongly pseudoconvex domain is Gromov hyperbolic.
For convex domains of finite type, the ideal and topological boundary also coincide:
Proposition 2.10.
[Zim16, Proposition 11.3, Proposition 11.5] Suppose is a bounded convex domain with finite type in the sense of D’Angelo. If are sequences such that and , then
In particular, the ideal boundary of is homeomorphic to the topological boundary of .
One important step in the proof of Theorem 1.4 is to show that the Gromov product is still reasonably behaved even when the domain does not have finite type:
Theorem 2.11.
(see Theorem 4.1 below) Suppose is a bounded convex domain with boundary and are sequences such that and .

If , then

If
then .
Although this behavior is much weaker than the finite type case, we can still use Theorem 2.11 to prove variants of Theorem 2.3 and Theorem 2.6. For instance, Theorem 2.11 and Karlsson’s [Kar01] work on the behavior of 1Lipschitz maps on general metric spaces imply the following:
Theorem 2.12.
(see Theorem 5.1 below) Suppose is a bounded convex domain with boundary. If is 1Lipschitz with respect to the Kobayashi metric, then either has a fixed point in or there exists some so that
for all .
Using Theorem 2.12 we can characterize the automorphisms of into elliptic, hyperbolic, and parabolic elements. Suppose is a bounded convex domain with boundary and . Then by Theorem 2.12 either has a fixed point in or there exists a complex supporting hyperplane of so that
for all . In this latter case, we call the attracting hyperplane of .
Definition 2.14.
Suppose is a bounded convex domain with boundary and . Then:

is elliptic if has a fixed point in ,

is parabolic if has no fixed point in and ,

is hyperbolic if has no fixed points in and . In this case we call the repelling hyperplane of .
Remark 2.15.
Notice that Theorem 2.12 implies that every automorphism of is either elliptic, hyperbolic, or parabolic.
Theorem 2.16.
(see Theorem 6.1 below) Suppose is a bounded convex domain with boundary. If and are neighborhoods of , in so that , then there exists a compact set with the following property: if is a geodesic with and , then .
2.3. Outline of the proof of Theorem 1.4
The difficult direction of Theorem 1.4 is to show that is biholomorphic to a polynomial ellipsoid when the limit set intersects at least two different closed complex faces. The main steps in the proof of this direction are the following:

Use another rescaling argument to show that the entire boundary has finite type (see Section 10).

Apply Bedford and Pinchuk’s result to deduce that is biholomorphic to a polynomial ellipsoid.
2.4. Other applications of Theorem 2.11
In this subsection we describe some other applications of Theorem 2.11 (many of which are used in the proof of Theorem 1.4).
Boundary extensions
A convex domain is called strictly convex if every supporting complex hyperplane intersects at exactly one point. When is this is equivalent to for every . For strictly convex domains we have the following corollary of Theorem 2.11:
Corollary 2.17.
Suppose is a bounded strictly convex domain with boundary. If are sequences such that and , then
As an application of this corollary we will prove the following result about boundary extensions:
Theorem 2.18.
(see Theorem 12.1 below) Suppose and are bounded convex domains with boundaries. If is strictly convex, then every isometric embedding extends to a continuous map .
In particular:
Corollary 2.19.
Suppose are bounded convex domains with boundaries. If is strictly convex, then every biholomorphism extends to a continuous map .
Corollary 2.20.
Suppose is a bounded strictly convex domain with boundary. Then every complex geodesic extends to a continuous map .
Remark 2.21.

As mentioned above, Theorem 2.18 is a consequence of the behavior of the Gromov product on convex domains. It is worth mentioning that Forstnerič and Rosay [FR87, Theorem 1.1] also used the behavior of the Gromov product (without using this terminology) to establish continuous boundary extensions of proper holomorphic maps between strongly pseudoconvex domains.

There is also an alternative approach to proving boundary extensions of holomorphic maps which uses lower bounds on the infinitesimal Kobayashi metric and a HardyLittlewood type lemma, see for instance [DF79, CHL88, Mer93, Bha16]. These arguments only appear to work when the infinitesimal Kobayashi metric obeys some estimate of the form
where , see [BZ16]. It is possible to construct smoothly bounded strictly convex domains where the infinitesimal Kobayash metric fails to satisfy such estimates.
Nonexistence of holomorphic maps
We will also use Theorem 4.1 to show that certain holomorphic maps cannot exist:
Theorem 2.22.
(see Theorem 9.7 below) Suppose is a bounded convex domain with boundary. Then there does not exist a holomorphic map which induces an isometric embedding .
Remark 2.23.

If is a Gromov hyperbolic metric space then there does not exist an isometric embedding of into . In particular, the above Corollary shows that convex domains with boundary have some hyperbolic behavior.
The GreeneKrantz conjecture
The second main step in the proof of Theorem 1.4 is related to an old conjecture of Greene and Krantz. In particular, in the 1990’s Greene and Krantz conjectured:
Conjecture 2.24.
[GK93] Suppose that is a bounded pseudoconvex domain with boundary. If , then has finite type in the sense of Kohn/D’Angelo/Catlin.
There are a number of partial results supporting the conjecture, see for instance the survey paper [Kra13]. In Section 9, we will prove the following special case of this conjecture:
Theorem 2.25.
(see Theorem 9.1 below) Suppose is a bounded convex domain with boundary. If there exists , , , and so that
then has finite type in the sense of D’Angelo.
Here is the idea of the proof: if had infinite type, then we could use a rescaling argument to construct a holomorphic map having (essentially) the properties in the hypothesis of Theorem 2.22 which is impossible.
3. Preliminaries
3.1. Notations

For let be the standard Euclidean norm and be the standard Euclidean distance.

Given an open set , , and let
and
3.2. The Kobayashi metric
Given a domain the (infinitesimal) Kobayashi metric is the pseudoFinsler metric
By a result of Royden [Roy71, Proposition 3] the Kobayashi metric is an upper semicontinuous function on . In particular if is an absolutely continuous curve (as a map ), then the function
is integrable and we can define the length of to be
One can then define the Kobayashi pseudodistance to be
This definition is equivalent to the standard definition by a result of Venturini [Ven89, Theorem 3.1].
One important property of the Kobayashi metric on a convex set is the following:
Proposition 3.1.
[Bar80] Suppose is a convex domain Then the following are equivalent:

is a Cauchy complete geodesic metric space,

does not contain any complex affine lines.
3.3. The disk and the upper half plane
For the disk and upper half plane the Kobayashi metric coincides with the Poincaré metric.
Let . Then
and
Next let . Then
and
3.4. Almost geodesics
In the proof of Theorem 1.4 it will often be convenient to work with a class of curves which we call almostgeodesics:
Definition 3.2.
Suppose is a metric space and is an interval.

A curve is a quasigeodesic if
for all .

If then a curve is an almostgeodesic if is an quasigeodesic and
for all .
The main motivation for considering almostgeodesics is Proposition 4.3 below which shows that inward pointing normal lines can be parametrized to be an almostgeodesics for convex domains with boundary. We should also note that an almostgeodesic is a geodesic, thus motivating the choice of additive factor.
4. The Gromov product
In this section we prove Theorem 2.11 which we restate:
Theorem 4.1.
Suppose is a bounded convex domain with boundary and are sequences such that and .

If , then

If
then .
We begin by proving a series of lemmas.
Lemma 4.2.
Suppose is a convex domain and is a complex hyperplane such that . Then for any we have
Proof.
Since is convex, there exists a real hyperplane so that and . By translating and rotating , we may assume that
and
Consider the projection given by . Then
and so
Now for we have
So
Since this implies the lemma. ∎
Suppose is a domain with boundary. If let be the inward pointing normal unit vector at .
Proposition 4.3.
Suppose is a bounded convex domain with boundary. Then there exists and so that if then the curve given by
is an almostgeodesic.
Remark 4.4.
The most difficult inequality to establish in the above proposition is the upper bound
To show this we will closely follow the proof of Proposition 2.5 in [FR87].
Proof.
For let
For let be the map
Since is we can pick so that for all .
Next let be a domain with boundary, , and symmetric about the real axis. Such a domain can be obtained by smoothing near the two corner points. Now since is symmetric about the real axis there exists a biholomorphic map with and
Since has boundary, extends to a diffeomorphism (see for instance [Gol69, page 426 Theorem 6]).
Now fix and so that
for .
Then for we have
Thus if we have:
On the other hand,
Thus for any the curve is a quasigeodesic.
Now since is , by possibly decreasing we can assume that
for all . This implies that there exists a so that
for all and . Then
and so
Thus is a quasigeodesic.
Thus for all the curve is an almostgeodesic with . ∎
Lemma 4.5.
Suppose is a bounded convex domain with boundary, , and . Then there exists and such that
when , , and .
Remark 4.6.
Abate [Aba89, Proposition 2.4.24, Corollary 2.4.25] proved a weaker version of the above lemma assuming that has boundary and .
Proof.
For a set and let
For a point let and for a point let
Since is only , we may have for arbitrarily close to .
Next for let
and
Notice that and are compact.
We claim that there exists so that . Suppose not, then for each there exists , , , and so that
Fix
and pass to a subsequence so that , , and . By construction and . So and . Thus
which contradicts the fact that is convex, is , and . So we can pick so that .
Now since and are compact there exists so that
Now let . Suppose that and . Then and . Moreover, if we pick and then
and
Now let be a geodesic with and . Since
there exists some so that . Then by Lemma 4.2 we have