Random complex dynamics andsemigroups of holomorphic maps 1footnote 11footnote 1Published in Proc. London Math. Soc. (2011), 102 (1), 50–112. 2000 Mathematics Subject Classification. 37F10, 30D05. Keywords: Random dynamical systems, random complex dynamics, random iteration, Markov process, rational semigroups, polynomial semigroups, Julia sets, fractal geometry, cooperation principle, noise-induced order.

# Random complex dynamics and semigroups of holomorphic maps 1

## Abstract

We investigate the random dynamics of rational maps on the Riemann sphere and the dynamics of semigroups of rational maps on We show that regarding random complex dynamics of polynomials, in most cases, the chaos of the averaged system disappears, due to the cooperation of the generators. We investigate the iteration and spectral properties of transition operators. We show that under certain conditions, in the limit stage, “singular functions on the complex plane” appear. In particular, we consider the functions which represent the probability of tending to infinity with respect to the random dynamics of polynomials. Under certain conditions these functions are complex analogues of the devil’s staircase and Lebesgue’s singular functions. More precisely, we show that these functions are continuous on and vary only on the Julia sets of associated semigroups. Furthermore, by using ergodic theory and potential theory, we investigate the non-differentiability and regularity of these functions. We find many phenomena which can hold in the random complex dynamics and the dynamics of semigroups of rational maps, but cannot hold in the usual iteration dynamics of a single holomorphic map. We carry out a systematic study of these phenomena and their mechanisms.

## 1 Introduction

In this paper, we investigate the random dynamics of rational maps on the Riemann sphere and the dynamics of rational semigroups (i.e., semigroups of non-constant rational maps where the semigroup operation is functional composition) on We see that the both fields are related to each other very deeply. In fact, we develop both theories simultaneously.

One motivation for research in complex dynamical systems is to describe some mathematical models on ethology. For example, the behavior of the population of a certain species can be described by the dynamical system associated with iteration of a polynomial such that preserves the unit interval and the postcritical set in the plane is bounded (cf. [7]). However, when there is a change in the natural environment, some species have several strategies to survive in nature. From this point of view, it is very natural and important not only to consider the dynamics of iteration, where the same survival strategy (i.e., function) is repeatedly applied, but also to consider random dynamics, where a new strategy might be applied at each time step. The first study of random complex dynamics was given by J. E. Fornaess and N. Sibony ([9]). For research on random complex dynamics of quadratic polynomials, see [2, 3, 4, 5, 6, 10]. For research on random dynamics of polynomials (of general degrees) with bounded planar postcritical set, see the author’s works [35, 34, 36, 37, 38, 39].

The first study of dynamics of rational semigroups was conducted by A. Hinkkanen and G. J. Martin ([13]), who were interested in the role of the dynamics of polynomial semigroups (i.e., semigroups of non-constant polynomial maps) while studying various one-complex-dimensional moduli spaces for discrete groups, and by F. Ren’s group ([11]), who studied such semigroups from the perspective of random dynamical systems. Since the Julia set of a finitely generated rational semigroup has “backward self-similarity,” i.e., (see Lemma 4.1 and [26, Lemma 1.1.4]), the study of the dynamics of rational semigroups can be regarded as the study of “backward iterated function systems,” and also as a generalization of the study of self-similar sets in fractal geometry.

For recent work on the dynamics of rational semigroups, see the author’s papers [26][39], [41], and [25, 42, 43, 44, 45].

In order to consider the random dynamics of a family of polynomials on , let be the probability of tending to starting with the initial value In this paper, we see that under certain conditions, the function is continuous on and has some singular properties (for instance, varies only on a thin fractal set, the so-called Julia set of a polynomial semigroup), and this function is a complex analogue of the devil’s staircase (Cantor function) or Lebesgue’s singular functions (see Example 6.2, Figures 2, 3, and 4). Before going into detail, let us recall the definition of the devil’s staircase (Cantor function) and Lebesgue’s singular functions. Note that the following definitions look a little bit different from those in [46], but it turns out that they are equivalent to those in [46].

###### Definition 1.1 ([46]).

Let be the unique bounded function which satisfies the following functional equation:

 12φ(3x)+12φ(3x−2)≡φ(x), φ|(−∞,0]≡0, φ|[1,+∞)≡1. (1)

The function is called the devil’s staircase (or Cantor function).

###### Remark 1.2.

The above is continuous on and varies precisely on the Cantor middle third set. Moreover, it is monotone (see Figure 1).

###### Definition 1.3 ([46]).

Let be a constant. We denote by the unique bounded function which satisfies the following functional equation:

 aψa(2x)+(1−a)ψa(2x−1)≡ψa(x), ψa|(−∞,0]≡0, ψa|[1,+∞)≡1. (2)

For each with , the function is called Lebesgue’s singular function with respect to the parameter

###### Remark 1.4.

The function is continuous on , monotone on , and strictly monotone on . Moreover, if , then for almost every with respect to the one-dimensional Lebesgue measure, the derivative of at is equal to zero (see Figure 1). For the details on the devil’s staircase and Lebesgue’s singular functions and their related topics, see [46, 12].

These singular functions defined on can be redefined by using random dynamical systems on as follows. Let and we consider the random dynamical system (random walk) on such that at every step we choose with probability and with probability We set We denote by the probability of tending to starting with the initial value Then, we can see that the function is equal to the devil’s staircase.

Similarly, let and let be a constant. We consider the random dynamical system on such that at every step we choose the map with probability and the map with probability Let be the probability of tending to starting with the initial value Then, we can see that the function is equal to Lebesgue’s singular function with respect to the parameter

We remark that in most of the literature, the theory of random dynamical systems has not been used directly to investigate these singular functions on the interval, although some researchers have used it implicitly.

One of the main purposes of this paper is to consider the complex analogue of the above story. In order to do that, we have to investigate the independent and identically-distributed (abbreviated by i.i.d.) random dynamics of rational maps and the dynamics of semigroups of rational maps on simultaneously. We develop both the theory of random dynamics of rational maps and that of the dynamics of semigroups of rational maps. The author thinks this is the best strategy since when we want to investigate one of them, we need to investigate the other.

To introduce the main idea of this paper, we let be a rational semigroup and denote by the Fatou set of , which is defined to be the maximal open subset of where is equicontinuous with respect to the spherical distance on . We call the Julia set of The Julia set is backward invariant under each element , but might not be forward invariant. This is a difficulty of the theory of rational semigroups. Nevertheless, we “utilize” this as follows. The key to investigating random complex dynamics is to consider the following kernel Julia set of , which is defined by This is the largest forward invariant subset of under the action of Note that if is a group or if is a commutative semigroup, then However, for a general rational semigroup generated by a family of rational maps with , it may happen that (see subsection 3.5, section 6).

Let Rat be the space of all non-constant rational maps on the Riemann sphere , endowed with the distance which is defined by , where denotes the spherical distance on Let Rat be the space of all rational maps with Let be the space of all polynomial maps with Let be a Borel probability measure on Rat with compact support. We consider the i.i.d. random dynamics on such that at every step we choose a map according to Thus this determines a time-discrete Markov process with time-homogeneous transition probabilities on the phase space such that for each and each Borel measurable subset of , the transition probability of the Markov process is defined as Let be the rational semigroup generated by the support of Let be the space of all complex-valued continuous functions on endowed with the supremum norm. Let be the operator on defined by This is called the transition operator of the Markov process induced by For a topological space , let be the space of all Borel probability measures on endowed with the topology induced by the weak convergence (thus in if and only if for each bounded continuous function ). Note that if is a compact metric space, then is compact and metrizable. For each , we denote by supp the topological support of Let be the space of all Borel probability measures on such that supp is compact. Let be the dual of . This can be regarded as the “averaged map” on the extension of (see Remark 2.21). We define the “Julia set” of the dynamics of as the set of all elements satisfying that for each neighborhood of , is not equicontinuous on (see Definition 2.17). For each sequence , we denote by the set of non-equicontinuity of the sequence with respect to the spherical distance on This is called the Julia set of Let

We prove the following theorem.

###### Theorem 1.5 (Cooperation Principle I, see Theorem 3.14 and Proposition 4.7).

Let Suppose that Then Moreover, for -a.e. , the -dimensional Lebesgue measure of is equal to zero.

This theorem means that if all the maps in the support of cooperate, the set of sensitive initial values of the averaged system disappears. Note that for any , Thus the above result deals with a phenomenon which can hold in the random complex dynamics but cannot hold in the usual iteration dynamics of a single rational map with

From the above result and some further detailed arguments, we prove the following theorem. To state the theorem, for a , we denote by the space of all finite linear combinations of unitary eigenvectors of , where an eigenvector is said to be unitary if the absolute value of the corresponding eigenvalue is equal to one. Moreover, we set Under the above notations, we have the following.

###### Theorem 1.6 (Cooperation Principle II: Disappearance of Chaos, see Theorem 3.15).

Let Suppose that and . Then we have all of the following statements.

• There exists a direct decomposition . Moreover, and is a closed subspace of Moreover, there exists a non-empty -invariant compact subset of with finite topological dimension such that for each , in as . Furthermore, each element of is locally constant on . Therefore each element of is a continuous function on which varies only on the Julia set

• For each , there exists a Borel subset of with with the following property.

• For each , there exists a number such that as , where diam denotes the diameter with respect to the spherical distance on , and denotes the ball with center and radius

• There exists at least one and at most finitely many minimal sets for , where we say that a non-empty compact subset of is a minimal set for if is minimal in with respect to inclusion.

• Let be the union of minimal sets for . Then for each there exists a Borel subset of with such that for each , as

This theorem means that if all the maps in the support of cooperate, the chaos of the averaged system disappears. Theorem 1.6 describes new phenomena which can hold in random complex dynamics but cannot hold in the usual iteration dynamics of a single For example, for any , if we take a point , where denotes the Julia set of the semigroup generated by , then for any ball with , expands as , and we have infinitely many minimal sets (periodic cycles) of

In Theorem 3.15, we completely investigate the structure of and the set of unitary eigenvalues of (Theorem 3.15). Using the above result, we show that if and int where int denotes the set of interior points, then has infinitely many connected components (Theorem 3.15-20). Thus the random complex dynamics can be applied to the theory of dynamics of rational semigroups. The key to proving Theorem 1.6 (Theorem 3.15) is to show that for almost every with respect to and for each compact set contained in a connected component of , as This is shown by using careful arguments on the hyperbolic metric of each connected component of Combining this with the decomposition theorem on “almost periodic operators” on Banach spaces from [18], we prove Theorem 1.6 (Theorem 3.15).

Considering these results, we have the following natural question: “When is the kernel Julia set empty?” Since the kernel Julia set of is forward invariant under , Montel’s theorem implies that if is a Borel probability measure on with compact support, and if the support of contains an admissible subset of (see Definition 3.54), then (Lemma 3.56). In particular, if the support of contains an interior point with respect to the topology of , then (Lemma 3.52). From this result, it follows that for any Borel probability measure on with compact support, there exists a Borel probability measure with finite support, such that is arbitrarily close to , such that the support of is arbitrarily close to the support of , and such that (Proposition 3.57). The above results mean that in a certain sense, for most Borel probability measures on Summarizing these results we can state the following.

###### Theorem 1.7 (Cooperation Principle III, see Lemmas 3.52, 3.56, Proposition 3.57).

Let be endowed with the topology such that in if and only if (a) for each bounded continuous function on , and (b) suppsupp with respect to the Hausdorff metric. We set and . Then we have all of the following.

• and are dense in .

• If the interior of the support of is not empty with respect to the topology of , then

• For each , the chaos of the averaged system of the Markov process induced by disappears (more precisely, all the statements in Theorems 1.5, 1.6 hold).

In the subsequent paper [40], we investigate more detail on the above result (some results of [40] are announced in [41]).

We remark that in 1983, by numerical experiments, K. Matsumoto and I. Tsuda ([20]) observed that if we add some uniform noise to the dynamical system associated with iteration of a chaotic map on the unit interval , then under certain conditions, the quantities which represent chaos (e.g., entropy, Lyapunov exponent, etc.) decrease. More precisely, they observed that the entropy decreases and the Lyapunov exponent turns negative. They called this phenomenon “noise-induced order”, and many physicists have investigated it by numerical experiments, although there has been only a few mathematical supports for it.

Moreover, in this paper, we introduce “mean stable” rational semigroups in subsection 3.6. If is mean stable, then and a small perturbation of is still mean stable. We show that if is a compact subset of Rat and if the semigroup generated by is semi-hyperbolic (see Definition 2.12) and , then there exists a neighborhood of in the space of non-empty compact subset of Rat such that for each , the semigroup generated by is mean stable, and

By using the above results, we investigate the random dynamics of polynomials. Let be a Borel probability measure on with compact support. Suppose that and the smallest filled-in Julia set (see Definition 3.19) of is not empty. Then we show that the function of probability of tending to belongs to and is not constant (Theorem 3.22). Thus is non-constant and continuous on and varies only on Moreover, the function is characterized as the unique Borel measurable bounded function which satisfies , and , where denotes the connected component of the Fatou set of containing (Proposition 3.26). From these results, we can show that has a kind of “monotonicity,” and applying it, we get information regarding the structure of the Julia set of (Theorem 3.31). We call the function a devil’s coliseum, especially when int (see Example 6.2, Figures 2, 3, and 4). Note that for any , is not continuous at any point of Thus the above results deal with a phenomenon which can hold in the random complex dynamics, but cannot hold in the usual iteration dynamics of a single polynomial.

It is a natural question to ask about the regularity of non-constant (e.g., ) on the Julia set For a rational semigroup , we set , where the closure is taken in , and we say that is hyperbolic if If is generated by as a semigroup, we write We prove the following theorem.

###### Theorem 1.8 (see Theorem 3.82 and Theorem 3.84).

Let and let . Let . Let with Let Suppose that for each with and suppose also that is hyperbolic. Then we have all of the following statements.

• , int, and , where denotes the Hausdorff dimension with respect to the spherical distance on

• Suppose further that at least one of the following conditions (a)(b)(c) holds.

• is bounded in .

Then there exists a non-atomic “invariant measure” on with supp and an uncountable dense subset of with and , such that for every and for each non-constant , the pointwise Hölder exponent of at , which is defined to be

 inf{α∈R∣limsupy→z|φ(y)−φ(z)||y−z|α=∞},

is strictly less than and is not differentiable at (Theorem 3.82).

• In (2) above, the pointwise Hölder exponent of at can be represented in terms of and the integral of the sum of the values of the Green’s function of the basin of for the sequence at the finite critical points of (Theorem 3.82).

• Under the assumption of (2), for almost every point with respect to the -dimensional Hausdorff measure where , the pointwise Hölder exponent of a non-constant at can be represented in terms of the and the derivatives of (Theorem 3.84).

Combining Theorems 1.5, 1.6, 1.8, it follows that under the assumptions of Theorem 1.8, the chaos of the averaged system disappears in the “sense”, but it remains in the “sense”. From Theorem 1.8, we also obtain that if is small enough, then for almost every with respect to and for each , is differentiable at and the derivative of at is equal to zero, even though a non-constant is not differentiable at any point of an uncountable dense subset of (Remark 3.86). To prove these results, we use Birkhoff’s ergodic theorem, potential theory, the Koebe distortion theorem and thermodynamic formalisms in ergodic theory. We can construct many examples of such that for each with where , is hyperbolic, , and possesses non-constant elements (e.g., ) for any (see Proposition 6.1, Example 6.2, Proposition 6.3, Proposition 6.4, and Remark 6.6).

We also investigate the topology of the Julia sets of sequences , where is a Borel probability measure on with compact support. We show that if is not bounded in , then for almost every sequence with respect to , the Julia set of has uncountably many connected components (Theorem 3.38). This generalizes [2, Theorem 1.5] and [4, Theorem 2.3]. Moreover, we show that if and only if , and that if , then for almost every with respect to , the -dimensional Lebesgue measure of filled-in Julia set (see Definition 3.40) of is equal to zero and has uncountably many connected components (Theorem 3.41 and Example 3.59). These results generalize [4, Theorem 2.2] and one of the statements of [2, Theorem 2.4].

Another matter of considerable interest is what happens when We show that if is a Borel probability measure on Rat with compact support and is “semi-hyperbolic” (see Definition 2.12), then if and only if (Theorem 3.71). We define several types of “smaller Julia sets” of . We denote by the “pointwise Julia set” of restricted to (see Definition 3.44). We show that if is semi-hyperbolic, then (Theorem 3.71). Moreover, if , is semi-hyperbolic, and , then (Theorem 3.71). Thus the dual of the transition operator of the Markov process induced by can detect the Julia set of To prove these results, we utilize some observations concerning semi-hyperbolic rational semigroups that may be found in [29, 32]. In particular, the continuity of is required. (This is non-trivial, and does not hold for an arbitrary rational semigroup.)

Moreover, even when , it is shown that if is included in the unbounded component of the complement of the intersection of the set of non-semi-hyperbolic points of and , then for almost every with respect to , the -dimensional Lebesgue measure of the Julia set of is equal to zero (Theorem 3.48). To prove this result, we again utilize observations concerning the kernel Julia set of , and non-constant limit functions must be handled carefully (Lemmas 4.6, 5.32 and 5.33).

As pointed out in the previous paragraphs, we find many new phenomena which can hold in random complex dynamics and the dynamics of rational semigroups, but cannot hold in the usual iteration dynamics of a single rational map. These new phenomena and their mechanisms are systematically investigated.

In the proofs of all results, we employ the skew product map associated with the support of (Definition 3.46), and some detailed observations concerning the skew product are required. It is a new idea to use the kernel Julia set of the associated semigroup to investigate random complex dynamics. Moreover, it is both natural and new to combine the theory of random complex dynamics and the theory of rational semigroups. Without considering the Julia sets of rational semigroups, we are unable to discern the singular properties of the non-constant finite linear combinations (e.g., , a devil’s coliseum) of the unitary eigenvectors of .

In section 2, we give some fundamental notations and definitions. In section 3, we present the main results of this paper. In section 4, we introduce the basic tools used to prove the main results. In section 5, we provide the proofs of the main results. In section 6, we give many examples to which the main results are applicable.

In the subsequent paper [40], we investigate the stability and bifurcation of (some results of [40] are announced in [41]).

Acknowledgment: The author thanks Rich Stankewitz for valuable comments. This work was supported by JSPS Grant-in-Aid for Scientific Research(C) 21540216.

## 2 Preliminaries

In this section, we give some basic definitions and notations on the dynamics of semigroups of holomorphic maps and the i.i.d. random dynamics of holomorphic maps.

Notation: Let be a metric space, a subset of , and . We set Moreover, for a subset of , we set Moreover, for any topological space and for any subset of , we denote by int the set of all interior points of

###### Definition 2.1.

Let be a metric space. We set endowed with the compact-open topology. Moreover, we set endowed with the relative topology from Furthermore, we set When is compact, we endow with the supremum norm Moreover, for a subset of , we set

###### Definition 2.2.

Let be a complex manifold. We set endowed with the compact open topology. Moreover, we set endowed with the compact open topology.

###### Remark 2.3.

, , , and are semigroups with the semigroup operation being functional composition.

###### Definition 2.4.

A rational semigroup is a semigroup generated by a family of non-constant rational maps on the Riemann sphere with the semigroup operation being functional composition([13, 11]). A polynomial semigroup is a semigroup generated by a family of non-constant polynomial maps. We set Rat : endowed with the distance which is defined by , where denotes the spherical distance on Moreover, we set endowed with the relative topology from Rat. Furthermore, we set endowed with the relative topology from Rat.

###### Definition 2.5.

Let be a compact metric space and let be a subsemigroup of The Fatou set of is defined to be s.t. is equicontinuous on (For the definition of equicontinuity, see [1].) The Julia set of is defined to be If is generated by , then we write If is generated by a subset of , then we write For finitely many elements , we set and . For a subset of , we set and We set , where Id denotes the identity map.

By using the method in [13, 11], it is easy to see that the following lemma holds.

###### Lemma 2.6.

Let be a compact metric space and let be a subsemigroup of Then for each , and Note that the equality does not hold in general.

The following is the key to investigating random complex dynamics.

###### Definition 2.7.

Let be a compact metric space and let be a subsemigroup of We set This is called the kernel Julia set of

###### Remark 2.8.

Let be a compact metric space and let be a subsemigroup of (1) is a compact subset of (2) For each , (3) If is a rational semigroup and if , then int (4) If is generated by a single map or if is a group, then However, for a general rational semigroup , it may happen that (see subsection 3.5 and section 6).

The following postcritical set is important when we investigate the dynamics of rational semigroups.

###### Definition 2.9.

For a rational semigroup , let where the closure is taken in This is called the postcritical set of

###### Remark 2.10.

If and , then From this one may know the figure of , in the finitely generated case, using a computer.

###### Definition 2.11.

Let be a rational semigroup. Let be a positive integer. We denote by the set of points satisfying that there exists a positive number such that for each , , for each connected component of Moreover, we set

###### Definition 2.12.

Let be a rational semigroup. We say that is hyperbolic if We say that is semi-hyperbolic if

###### Remark 2.13.

We have If is hyperbolic, then is semi-hyperbolic.

It is sometimes important to investigate the dynamics of sequences of maps.

###### Definition 2.14.

Let be a compact metric space. For each and each with , we set and we set

 Fγ:={z∈Y∣∃ neighborhood U of z s.t. {γn,1}n∈N is equicontinuous on U}

and The set is called the Fatou set of the sequence and the set is called the Julia set of the sequence

###### Remark 2.15.

Let and let . Then by [1, Theorem 2.8.2], Moreover, if is a non-empty compact subset of and , then by [29], is a perfect set and has uncountably many points.

We now give some notations on random dynamics.

###### Definition 2.16.

For a topological space , we denote by the space of all Borel probability measures on endowed with the topology such that in if and only if for each bounded continuous function , Note that if is a compact metric space, then is a compact metric space with the metric , where is a dense subset of Moreover, for each , we set Note that is a closed subset of Furthermore, we set

For a complex Banach space , we denote by the space of all continuous complex linear functionals , endowed with the weak topology.

For any , we will consider the i.i.d. random dynamics on such that at every step we choose a map according to (thus this determines a time-discrete Markov process with time-homogeneous transition probabilities on the phase space such that for each and each Borel measurable subset of , the transition probability of the Markov process is defined as ).

###### Definition 2.17.

Let be a compact metric space. Let

1. We set (thus is a closed subset of ). Moreover, we set endowed with the product topology. Furthermore, we set This is the unique Borel probability measure on such that for each cylinder set in , We denote by the subsemigroup of generated by the subset of

2. Let be the operator on defined by is called the transition operator of the Markov process induced by Moreover, let be the dual of , which is defined as for each and each Remark: we have and for each and each open subset of , we have

3. We denote by the set of satisfying that there exists a neighborhood of in such that the sequence is equicontinuous on We set

4. We denote by the set of satisfying that the sequence is equicontinuous at the one point We set

We have and

###### Remark 2.19.

Let be a closed subset of Rat. Then there exists a such that