Fixed subgroups of automorphisms of relatively hyperbolic groups

# Fixed subgroups of automorphisms of relatively hyperbolic groups

Ashot Minasyan School of Mathematics, University of Southampton, Highfield, Southampton, SO17 1BJ, United Kingdom.  and  Denis Osin Department of Mathematics, Vanderbilt University, Nashville TN 37240, USA.
###### Abstract.

Let be a finitely generated relatively hyperbolic group. We show that if no peripheral subgroup of is hyperbolic relative to a collection of proper subgroups, then the fixed subgroup of every automorphism of is relatively quasiconvex. It follows that the fixed subgroup is itself relatively hyperbolic with respect to a natural family of peripheral subgroups. If all peripheral subgroups of are slender (respectively, slender and coherent), our result implies that the fixed subgroup of every automorphism of is finitely generated (respectively, finitely presented). In particular, this happens when is a limit group, and thus for any , is a limit subgroup of .

###### Key words and phrases:
Relatively hyperbolic groups, fixed subgroups of automorphisms.
###### 2000 Mathematics Subject Classification:
20F67, 20E36, 20F65
The second author was supported by the NSF grants DMS-0605093, DMS-1006345, and by the RFBR grant 05-01-00895.

## 1. Introduction

Given a group and an automorphism , let denote the fixed subgroup of , i.e.,

 Fix(φ)={g∈G∣φ(g)=g}.

Gersten [10] proved that if is a finitely generated free group, then is finitely generated for every . Collins and Turner [5] generalized this result by showing that has a finite Kuroš decomposition provided is a finite free product of freely indecomposable groups. Another generalization of Gersten’s theorem was found by Neumann [15], who showed that for every word hyperbolic group and every , is quasiconvex in .

In this paper we study fixed subgroups of automorphisms of a more general class of groups, which includes hyperbolic groups as well as finitely generated free products of freely indecomposable groups. More precisely, we deal with finitely generated groups hyperbolic relative to NRH subgroups. Recall that a nontrivial group is called non-relatively hyperbolic (or NRH) if is not hyperbolic relative to any collection of proper subgroups. The class of NRH groups includes many examples of interest. Below we list just some of them.

1. Unconstricted groups (defined by C. Druţu and M. Sapir in [7]). Recall that a finitely generated group is unconstricted if some asymptotic cone of does not have cut points. Examples of unconstricted groups include direct products of infinite groups, non-virtually cyclic groups satisfying a nontrivial law (e.g., solvable groups and groups of finite exponent) [7], many lattices in higher rank semi-simple groups [8], etc.

2. Suppose that a group is generated by a set consisting of elements of infinite order. The corresponding commutativity graph has as the set of vertices, two of which are joined by an edge if the corresponding elements commute. Assume that some adjacent pair of vertices generates . Then is NRH [2]. For example, this class includes many constricted groups such as for , all but finitely many mapping class groups, and freely indecomposable right angled Artin groups.

3. Non-virtually cyclic groups with infinite center [14, Lemma 10.2].

4. Non-virtually cyclic groups which do not contain non-abelian free subgroups ([7, Prop. 6.5]). In particular, non-virtually cyclic amenable groups as well as various ‘monsters’.

In many cases when peripheral subgroups are relatively hyperbolic themselves, we can still get an NRH peripheral structure in the following way. Suppose that is hyperbolic relative to and every is hyperbolic relative to proper subgroups . Then is hyperbolic relative to ([7, Cor. 1.14]). We exclude trivial subgroups from and if some of the subgroups from are hyperbolic relative to proper subgroups, we repeat this step again. We say that the process terminates, if after some step we obtain a (possibly empty) collection of NRH peripheral subgroups.

Note that the above process may not terminate even for hyperbolic groups. Recall that any hyperbolic group is hyperbolic relative to any quasiconvex malnormal subgroup [4]. Thus any infinite sequence of quasiconvex malnormal subgroups leads to an infinite process. However, in this case there also exists an obvious process which does stop as is hyperbolic relative to the empty collection of subgroups. Behrstock, Druţu and Mosher showed that there exists a finitely generated group for which no such a process terminates [3, Proposition 6.3]). The question of whether for every finitely presented relatively hyperbolic group there exists a terminating process is still open.

Our main result is the following.

###### Theorem 1.1.

Let be a finitely generated group which is hyperbolic relative to a family of NRH subgroups. Then for every , the fixed subgroup is relatively quasiconvex in .

Since the intersection of any two relatively quasiconvex subgroups is relatively quasiconvex [17, Prop. 4.18], the same result holds for any finite collection of automorphisms.

In order to prove Theorem 1.1 we first show that any automorphism of a group , that is hyperbolic relative to a collection of NRH subgroups, respects the peripheral structure (see Definition 3.1). Although this observation is quite elementary, it plays an important role in our paper. It allows us to conclude that induces a quasiisometry of the relative Cayley graph and to use the geometric machinery of relatively hyperbolic groups partially developed in [17]. In fact, our proof of Theorem 1.1 would remain valid if instead of requiring the peripheral subgroups of to be NRH one demanded the automorphism to respect some peripheral structure on . It would be interesting to see whether the conclusion of Theorem 1.1 holds in general, without any of these two requirements.

Note that relative quasiconvexity of a subgroup is independent of the choice of the finite generating set for the group [17, Prop. 4.10], but may, in general, depend on the selection of the family of peripheral subgroups. Nevertheless, relatively quasiconvex subgroups are well-behaved and have many good properties. For instance, C. Hruska [13] proved that relatively quasiconvex subgroups of relatively hyperbolic groups are themselves relatively hyperbolic with a natural induced peripheral structure. This allows us to obtain some results about the algebraic structure of fixed subgroups.

###### Corollary 1.2.

Assume that is a finitely generated group which is hyperbolic relative to a family of NRH subgroups and . Let be the set of all conjugates of peripheral subgroups of and let

 Pφ={P∩Fix(φ)∣P∈Pand|P∩Fix(φ)|=∞}.

Then the action of on by conjugation has finitely many orbits and is hyperbolic relative to representatives of these orbits. In particular, if is empty, then is finitely generated and word hyperbolic.

Recall that a group is called slender if every subgroup of is finitely generated and is called coherent if every finitely generated subgroup of is finitely presented. Since a group hyperbolic relative to a finite family of finitely generated (respectively, finitely presented) peripheral subgroups is itself finitely generated (respectively, finitely presented), the next result easily follows from Corollary 1.2.

###### Corollary 1.3.

If a finitely generated group is hyperbolic relative to slender subgroups, then for every , is finitely generated. If, in addition, all peripheral subgroups of are coherent, then is finitely presented. In particular, the latter conclusion holds for finitely generated relatively hyperbolic groups with virtually polycyclic peripheral subgroups.

One particular application of the above corollary shows that for any automorphism of a limit group , is finitely generated, and thus is a limit group itself. Indeed, Dahmani [6] and, independently, Alibegović [1] proved that any limit group is hyperbolic relative to the collection of representatives of conjugacy classes of maximal abelian non-cyclic subgroups.

Neumann’s original motivation for showing that is quasiconvex when is a hyperbolic group [15] was the result of S. Gersten and H. Short [11, Thm. 2.2] that, for a regular language on a group , a subgroup is -rational if and only if is -quasiconvex. This result can be combined with Gromov’s theorem [12, 8.5], claiming that in any hyperbolic group the set of all geodesic words forms a regular language , to conclude that is rational with respect to . It is also known that rationality and quasiconvexity of a subset of a hyperbolic group are independent of the choice of an automatic structure on it [16].

In a relatively hyperbolic group the situation is more complicated. For example, let be the free product of a free abelian group of rank with an infinite cyclic group, and let be the automorphism interchanging and and sending to . Then is hyperbolic relative to and . It is easy to see that is a regular language on , which can be naturally extended (using normal forms in free products) to an automatic language on . However, in this case is not -quasiconvex, and thus will not be -rational.

Nonetheless, in the case when all of the peripheral subgroups of a relatively hyperbolic group are abelian and , Theorem 1.1 can be combined with a result of D. Rebbechi [18, Thm. 9.1] in order to conclude that is biautomatic.

The paper is structured as follows. In Section 2 we include the necessary background on relatively hyperbolic groups and relatively quasiconvex subgroups; in Section 3 we give several auxiliary definitions and prove a number of technical results, which will be employed in our proof of the main result in Section 4.

Acknowledgment. The results of this paper were obtained when the second author was stuck in the UK due to the eruption of the Eyjafjallajökull volcano in April 2010. We would like to thank Eyjafjallajökull for providing us with a great opportunity to work together. We also thank the anonymous referee for helpful remarks and suggestions.

## 2. Preliminaries

Notation. Given a group generated by a subset , we denote by the Cayley graph of with respect to and by the word length of an element . We always assume that generating sets are symmetrized, i.e., . If is a (simplicial) path in , denotes its label, denotes its length, and denote its starting and ending vertex respectively. The notation will be used for the path in obtained by traversing backwards. For a word , written in the alphabet , will denote its length. For two words and we shall write to denote the letter-by-letter equality between them.

### Relatively hyperbolic groups.

In this paper we use the notion of relative hyperbolicity which is sometimes called strong relative hyperbolicity and goes back to Gromov [12]. There are many equivalent definitions of (strongly) relatively hyperbolic group. We briefly recall one of them and refer the reader to [4, 7, 9, 13, 17] for details.

Let be a group, let be a collection of pairwise distinct subgroups of , and let be a subset of . We say that is a relative generating set of with respect to if is generated by together with the union of all . (In what follows we always assume to be symmetric, i.e., .) In this situation the group can be regarded as a quotient group of the free product

 (1) F=(∗λ∈ΛHλ)∗F(X),

where is the free group with the basis . If the kernel of the natural homomorphism is the normal closure of a subset in the group , we say that has relative presentation

 (2) ⟨X,Hλ,λ∈Λ|R⟩.

If and , the relative presentation (2) is said to be finite and the group is said to be finitely presented relative to the collection of subgroups .

Set

 (3) H=⨆λ∈Λ(Hλ∖{1}).

Given a word in the alphabet such that represents in , there exists an expression

 (4) WF=k∏i=1f−1iR±1ifi

with the equality in the group , where and for . The smallest possible number in a representation of the form (4) is called the relative area of and is denoted by .

###### Definition 2.1 (Relatively hyperbolic groups).

A group is hyperbolic relative to a collection of subgroups , called peripheral subgroups, if is finitely presented relative to and there is a constant such that for any word in representing the identity in , we have

 (5) Arearel(W)≤C∥W∥.

This definition is independent of the choice of the finite generating set and the finite set in (2) (see [17]). In particular, is an ordinary (Gromov) hyperbolic group if is hyperbolic relative to the empty family of peripheral subgroups.

###### Remark 2.2.

Note that, by definition, if and each subgroup , , is finitely generated [finitely presented], then is also finitely generated [resp., finitely presented].

Let be a group generated by a finite set and let be a subgroup generated by a finite set . Recall, that is said to be undistorted in , if there exists such that for every one has .

In general, the relatively hyperbolic group does not have to be finitely generated, and the collection of peripheral subgroups could be infinite. However, the second author proved the following:

###### Lemma 2.3 ([17], Thm. 1.1 and Lemma 5.4).

Let be a finitely generated relatively hyperbolic group. Then the collection of peripheral subgroups is finite, every peripheral subgroup is finitely generated and undistorted in .

###### Lemma 2.4 ([17], Thm. 1.4).

Let be a group hyperbolic relative to a collection of subgroups . Then the following conditions hold.

1. For every , , and every , we have .

2. For every and , we have .

### Relatively quasiconvex and undistorted subgroups.

The following definition was suggested in [17].

###### Definition 2.5 (Relatively quasiconvex subgroups).

Let be a group generated by a finite set and hyperbolic relative to a family of subgroups . A subgroup is called relatively quasiconvex with respect to (or simply relatively quasiconvex when the collection is fixed) if there exists a constant such that the following condition holds. For any and any geodesic path from to in , each vertex of satisfies .

We will need two results about relatively quasiconvex subgroups. The first one is established in [13, Thm. 9.1].

###### Lemma 2.6.

Let be a finitely generated relatively hyperbolic group, a relatively quasiconvex subgroup of . Let be the set of all conjugates of peripheral subgroups of and let

 Q={P∩K∣P∈Pand|P∩K|=∞}.

Then the action of on by conjugation has finitely many orbits and is hyperbolic relative to representatives of these orbits.

In [7, Thm. 1.8] Druţu and Sapir showed that the conclusion of Lemma 2.6 holds for every undistorted subgroup . Later Hruska proved the following in [13, Thm. 1.5]:

###### Lemma 2.7.

Let be a finitely generated group hyperbolic with respect to a collection of subgroups and let be a finitely generated undistorted subgroup of . Then is relatively quasiconvex. In particular, the conclusion of the previous lemma holds for .

Note that relative quasiconvexity does not, in general, imply that the subgroup is undistorted (indeed, by definition, any subgroup of a peripheral subgroup is relatively quasiconvex; however, may be distorted in , and hence in ).

### Components.

Let be a group hyperbolic relative to a family of subgroups . We recall some auxiliary terminology introduced in [17], which plays an important role in our paper.

###### Definition 2.8 (Components).

Let be a path in the Cayley graph . A (non-trivial) subpath of is called an -component (or simply a component), if the label of is a word in the alphabet , for some , and is not contained in a longer subpath of with this property. Two -components of paths , (respectively) in are called connected if there exists a path in that connects some vertex of to some vertex of , and is a word consisting of letters from . In algebraic terms this means that all vertices of and belong to the same coset for a certain . Note that we can always assume that has length at most , as every non-trivial element of is included in the set of generators. A component of a path is isolated if it is not connected with any other component of .

In what follows, let be a group hyperbolic relative to a collection of subgroups and generated by a finite set . Note that is finite in this case and every is finitely generated [17, Theorem 1.1].

Let and be real numbers and let be a path in . Recall that is said to be -quasigeodesic if for any subpath of we have . It is not difficult to see that a path that is a concatenation of a geodesic path with a path of length at most is -quasigeodesic.

Given a path in we denote by the -length of the element represented by the label of ; in other words, . Recall that a path in is called a path without backtracking if for any , every –component of is isolated. Evidently any geodesic path in is without backtracking. The following is a reformulation of Farb’s Bounded Coset Penetration property (cf. [9]) in terms of the relative Cayley graph (see [17, Theorem 3.23]).

###### Lemma 2.9.

For any , and , there exists a constant such that the following conditions hold. Let , be -quasigeodesics without backtracking in such that and .

1. Suppose that for some , is an -component of such that ; then there exists an –component of such that is connected to .

2. Suppose that for some , and are connected -components of and respectively. Then and .

A vertex of a path in is phase if it is not an inner vertex of some component of . Observe that every vertex of a geodesic segment is phase, because all components consist of single edges. It is well known that in a hyperbolic group quasigeodesics with same endpoints are uniformly close to each other. An analogue of this statement for relatively hyperbolic groups was established in [17, Prop. 3.15]:

###### Lemma 2.10.

For any , and there exists a constant having the following property. Let and be two -quasigeodesic paths in such that , and is without backtracking. Then for any phase vertex of there exists a phase vertex of such that .

## 3. Technical lemmas

###### Definition 3.1 (Respecting peripheral structure).

Let be a group hyperbolic relative to a family of peripheral subgroups and let . We will say that respects the peripheral structure of if for every there is such that is a conjugate of in .

Throughout the rest of the paper will denote a group generated by a finite set and hyperbolic relative to a collection of NRH subgroups . In particular, all peripheral subgroups of are infinite. Note also that by Lemma 2.3.

###### Lemma 3.2.

With the above assumptions on , every respects the peripheral structure of , i.e., for each there is a unique such that is a conjugate of in . Moreover, the map , is a bijection.

###### Proof.

By Lemma 2.3 every is finitely generated and undistorted in . Hence so is (because an automorphism is always a quasiisometry when the group is equipped with a word metric given by some finite generating set). By Lemma 2.7, is relatively hyperbolic and each of its peripheral subgroups is an intersection of with a conjugate of some . Since is an NRH group, it can be hyperbolic only relative to itself. Therefore for some and . If this inclusion is proper, then properly contains . Applying Lemma 2.7 one more time, we obtain that is relatively hyperbolic with respect to a collection of subgroups containing . Since is NRH, this is again impossible. Hence . If is also conjugate to for some , then by Lemma 2.4 as all peripheral subgroups are infinite. Repeating the same arguments for we obtain that is injective (and, hence, bijective) on . ∎

From now on we fix an automorphism . For each fix and so that . Since and relative hyperbolicity is independent of the choice of the finite generating set , we can further assume that for every . Finally we set

 (6) S=maxx∈X{|φ(x)|X,|φ−1(x)|X}.
###### Definition 3.3 (Image of a path).

For every we fix a shortest word in the alphabet that represents in . Let be an edge of labelled by some . By we denote the path from to constructed as follows. If , we define to be the path with label . If for some , , where . In this case we let to be the path of length with . Hence the middle edge of will be its -component; we will call it the companion of and denote by . Given a path in , where are edges of , the path will be called the image of , and will be denoted by . Note that and .

Since every component of a geodesic path in consists of a single edge, Definition 3.3 together with Lemma 3.2 and the fact that , for all , easily imply the following

###### Remark 3.4.

For a geodesic path in , each component of consists of a single edge.

###### Lemma 3.5.

(a) Let be an edge of labelled by a letter from . Then and .

(b) Let be edges of labelled by letters from . Then and are connected (i.e., there is such that , and vertices of and belong to the same left coset of ) if and only if and are connected.

###### Proof.

(a)  Recall that for every . Using the triangle inequality and (6) we obtain . Similarly

(b)  Let , , , . If and are connected, then , for some , and . Then and . Clearly and . Therefore .

Conversely, suppose that and are connected, i.e., , for some and . By Lemma 3.2, there is a unique such that . Thus and , . Consequently , implying that and are connected as well. ∎

###### Lemma 3.6.

(a) Suppose that is a path without backtracking in such that every component of is an edge. Then is a path without backtracking.

(b) For every , , there exists a constant such that for any -quasigeodesic in , is -quasigeodesic.

###### Proof.

To prove part (a) it suffices to note that every component of is a companion of some component of and two components of are connected if and only if their companions are connected by Lemma 3.5.

For proving part (b), observe that for every , and hence for all . Consider any subpath of . By definition, there is a subpath of such that is contained in and , . Therefore,

where . ∎

###### Definition 3.7 (Fine geodesics).

Let be a non-negative real number. A geodesic in will be called -fine if no component of , with , is connected to its companion.

An easy argument (see Lemma 4.3 in Section 4) shows that if a component of a geodesic segments , with , is connected to its companion, then its endpoints are close to . Therefore the rest this section is devoted to studying properties of -fine geodesics.

###### Lemma 3.8.

Consider arbitrary and , and set where is given by Lemma 2.9. Suppose and are two geodesic segments in with and , and is -fine. Then is -fine.

###### Proof.

Recall that a component of a geodesic path in is always a single edge. Assume, on the contrary, that there is a component of that is connected to its companion and . Since , part (a) of Lemma 2.9 implies that is connected to some component of , and part (b) together with the triangle inequality yield . Hence, according to the assumptions, is not connected with . On the other hand, and are connected by Lemma 3.5, and since “connectedness” is a symmetric and transitive relation, one can conclude that must be connected with , arriving to a contradiction. Thus is -fine. ∎

The following lemma establishes a sort of local finiteness for -fine geodesics.

###### Lemma 3.9.

For every there exists an increasing function such that the following holds. Let be an -fine geodesic in such that . Suppose that , where is a component of . Then .

###### Proof.

We will establish the claim by induction on . Note that is an -quasigeodesic without backtracking, where is the constant from Lemma 3.6. Let be the constant provided by Lemma 2.9. Set

 α(n)=(E+ε+2)(S+2ε+2)n+1 for all n∈N∪{0}.

If then for each and the claim holds. Otherwise, by Lemma 2.9, must be connected with a component of and by the assumptions. Thus there are two cases to consider.

Case 1. is a component of (in particular, this happens when ). Let be an edge connecting to and let denote the segment of from to . Note that the path is a -quasigeodesic in (and hence it is an -quasigeodesic) without backtracking, and the edge is a component of . If , then by Lemma 3.5; consequently for all .

Suppose, now, that , then Lemma 2.9, applied to the quasigeodesics and , implies that must be connected with some component of . And since is without backtracking (by Lemma 3.6) and , cannot be connected to . Hence is connected with a component of . In particular, , i.e., the base of induction () has already been established. By the induction hypothesis we have . On the other hand, by the second part of Lemma 2.9, and the triangle inequality gives . Combining these with the claim of Lemma 3.5, we obtain

 lX(e)≤S(lX(eφ)+2)≤S(lX(h)+2ε+2)≤S(α(n−1)+2ε+2)=(S+2ε+2)α(n−1)−(2ε+2)(α(n−1)−S)<α(n).

Case 2. is a component of . Note that by Lemma 2.9, therefore by the triangle inequality. Since every component of is the companion of some component of , for some component of . By induction, . Hence, recalling the statement of part (a) of Lemma 3.5, we obtain

 lX(e)≤lX(fφ)+2ε≤SlX(f)+2ε+2≤(S+2ε+2)α(n−1)=α(n).

Thus we have established the inductive step and finished the proof of the lemma. ∎

###### Definition 3.10 (The set I(x,E,R)).

Given , and , let denote the set of all geodesics in of length at most that are initial segments of -fine geodesic paths connecting with elements of .

The following corollary is an immediate consequence of Lemma 3.9:

###### Corollary 3.11.

For any , and , the set is finite. In particular, there exists such that for any and any one has .

(The fact that does not depend on follows from the fact that left translation by is a label-preserving automorphism of ).

###### Definition 3.12 (Large central component).

Consider any non-negative real number , and let be a geodesic triangle in with sides . We will say that has an -large central component if for each , contains a component , are pairwise connected and

 (7) lX(ai)>T=max{S(3ε0+2),E},

where is given by (6), and , are the constants from Lemmas 2.9 and 3.6, respectively. The edges will be called the sides of the -large central component.

###### Lemma 3.13.

Let be a non-negative real number and be a geodesic triangle in with an -large central component. Suppose that vertices of belong to . Then no side of is -fine; more precisely, every side of the -large central component is connected to its companion.

###### Proof.

Let denote the sides of (such that , where the indices are taken modulo ), and let be pairwise connected components of , respectively, satisfying (7). By part (b) of Lemma 3.5 the component of are also pairwise connected. Note also that (7) and part (a) of Lemma 3.5 imply that for . Hence by Lemmas 3.6 and 2.9, must be connected to a component of for each . If for some , then for all since all are connected and every component of a side of is isolated in that side (because the side is geodesic). Therefore no side of would be -fine. So we can assume that no coincides with . Below we show that this case is impossible by arriving at a contradiction.

For each , let and let denote the path (of length at most ) of connecting to (here and below indices are modulo ). The triangle is cut into a hexagon and triangles , (see Fig. 1). By our assumption, every component belongs to one of these triangles. Hence at least one of the triangles, say , contains exactly one of such components, say, . Again, since every component of a side of is isolated in that side, is an isolated component of . Note that and are -quasigeodesic paths without backtracking and with same endpoints, hence Lemma 2.9 implies that . However using part (2) of Lemma 2.9 and the triangle inequality we obtain , resulting in a contradiction. ∎

###### Definition 3.14 (Projections).

Let be a geodesic in and be any vertex. The projection of to is the set of vertices defined by

 prL(z)={g∈G∩L|dX∪H(z,g)=dX∪H(z,L)}.

The next statement is quite standard.

###### Lemma 3.15.

Consider three vertices in , a geodesic segment between and , any and any geodesic connecting with . Let where and are geodesic subpaths with , , . Then and are -quasigeodesic paths without backtracking.

###### Proof.

We will prove the statement for the path as the other case is symmetric. Consider any subpath of . The situations when both belong either to or to are trivial, therefore we can assume that and . Since and , we have . As and is geodesic we also have . The triangle inequality gives . Combining these inequalities together, we can conclude that . Therefore, applying the triangle inequality again, we achieve , as required.

The paths and are geodesic, and, hence, are without backtracking. Now, suppose a component of is connected to a component of . Then , but as shown in the previous paragraph. Therefore and thus , implying that . Consequently is a single component of , and so is without backtracking. ∎

###### Lemma 3.16.

For every there exists a constant such that the following holds. Let be a geodesic triangle in which does not contain an -large central component. Then there exists a vertex (where denotes the corresponding side of ) and vertices , such that .

###### Proof.

Let be the number from Definition 3.12, and let and be the constants given by Lemmas 2.9 and 2.10 respectively. Set .

Consider any vertex and any geodesic path