# The isomorphism problem for all hyperbolic groups.

###### Abstract

We give a solution to Dehn’s isomorphism problem for the class of all hyperbolic groups, possibly with torsion. We also prove a relative version for groups with peripheral structures. As a corollary, we give a uniform solution to Whitehead’s problem asking whether two tuples of elements of a hyperbolic group are in the same orbit under the action of . We also get an algorithm computing a generating set of the group of automorphisms of a hyperbolic group preserving a peripheral structure.

## 1 Introduction

In 1912, Dehn asked about three fundamental algorithmic problems for groups: the word problem, the conjugacy problem, and the isomorphism problem. The word problem and the conjugacy problem in a group consist in deciding algorithmically whether two words in some finite generating set represent the same or conjugate elements in . On the other hand, the isomorphism problem for a class of groups consists in deciding algorithmically whether two group presentations in this class represent isomorphic groups. It is remarkable that the answers to such algorithmic problems, positive or negative, in generality or in particular classes, have repeatedly revealed deep and fruitful structures in group theory.

In the 1950’s, it was discovered that all of these problems have negative answers in the class of all finitely presented groups. More precisely, Boone and Novikov proved the existence of a finitely presented group for which no algorithm can solve the word problem [Boone_WP, Novikov_WP]. Adyan and Rabin used such a group to prove that no algorithm can decide whether an arbitrary finite presentation defines a non-trivial group [Adyan_algorithmic, Rabin_recursive].

However there are many interesting and large classes of groups for which algorithms solving the word and conjugacy problems are well know. Therefore, groups with unsolvable word problem are often regarded as monsters, or as constructed “on purpose”.

In striking contrast, the isomorphism problem is unsolvable for some very natural classes of groups, including the class of free-by-free groups (Miller [Miller_decision]), the class of [free abelian]-by-free groups (Zimmermann [Zimmermann_klassifikation]), or the class of solvable groups of derived length (Baumslag-Gildenhuys-Strebel [BGS_algorithmically2] following [Kharlampovich_unsolvable81]).

In fact, until recently and the use of geometric group theory, the isomorphism problem was known to be decidable in only a few cases,
including notably the class of virtually polycyclic and nilpotent groups by Grunewald and Segal ([GruSe_nilpotent, Segal_decidable]).
The isomorphism problem for the class of Coxeter groups, the class of generalised Baumslag-Solitar groups, and of one-relator groups
have been investigated but remain unsettled [Bahls_isomorphism, ClFo_isomorphism, Pietrowski_isomorphism, Pride_isomorphism].

Z. Sela’s solution of the isomorphism problem for the class of rigid torsion-free hyperbolic groups was certainly a great breakthrough [Sela_isomorphism]. Sela also had a solution for the class of all torsion-free hyperbolic groups but did not publish it. This program was continued by D. Groves and the first author, who simplified Sela’s initial approach, and gave a proof for the class of all torsion free hyperbolic groups and toral relative hyperbolic groups [DaGr_isomorphism].

#### Statement of main results.

In this paper, we give a solution to the isomorphism problem for the class of all word-hyperbolic groups (as defined in [Gromov_hyperbolic]), possibly with torsion.

###### Theorem 1.

There is an explicit algorithm that takes as input two presentations of hyperbolic groups, and which decides whether these groups are isomorphic or not.

A result by Newman shows that one-relator groups with non-trivial torsion are hyperbolic [Newman_one-relator]. We thus get the following corollary:

###### Corollary 2.

The isomorphism problem for one-relator groups with non-trivial torsion is solvable.

In our methods, the solution of the isomorphism problem is symbiotic with the computation of a generating set of the group of automorphisms.

###### Theorem 3.

There is an explicit algorithm that takes as input a presentation of a hyperbolic group , and which computes a generating set of and .

In our solution of the isomorphism problem (as well as in [DaGr_isomorphism]), one needs to compute various decompositions of hyperbolic groups as amalgamated free products, HNN extensions, and more generally, as graphs of groups. This raises the natural question whether vertex groups of a graph of groups are isomorphic relative to their adjacent edge subgroups. A variation of this problem is to consider groups equipped with marked peripheral structures, namely a finite ordered collection of tuples of elements that are thought of as generating sets of the adjacent edge groups. More precisely, is a marked peripheral structure on if each is a tuple of elements of (each tuple being understood up to conjugacy).

Given two groups, and marked peripheral structures and , the marked isomorphism problem consists in deciding if there exists an isomorphism sending to a conjugate of for all .

###### Theorem 4.

[see Theorem LABEL:thm_whitehead_marque] The marked isomorphism problem is solvable among hyperbolic groups with marked peripheral structures.

Moreover, one can algorithmically compute a generating set of the group of automorphisms of a hyperbolic group with marked peripheral structure.

Although of more technical appearance, Theorem 4 has a particularly nice consequence. The Whitehead problem in a group , asks whether two tuples of elements are in the same orbit under the automorphism group of . Theorem 4 gives a uniform solution to the Whitehead problem for all hyperbolic groups.

###### Corollary 5 (see cor. LABEL:cor_whitehead_1234).

Given a hyperbolic group and , one can decide if there exists an automorphism sending to for all .

One can also decide if there exists an automorphism sending to a conjugate of for all .

A solution of the Whitehead problem was known for free groups [Whitehead_equivalent, HigLyn_Whitehead], and surface groups
[LeVo_Whitehead].
It is interesting to notice that even in these cases,
our approach is quite different from previous ones
as it is ultimately based on the computation of relative Grushko and JSJ decompositions
of .

#### Structural features.

Our approach follows the strategy initiated by Sela [Sela_isomorphism], and continued by Groves and the first author as exposed in [DaGr_isomorphism, Section 2].

Our result is based on the following three main structural features: {itemize*}

the Stallings-Dunwoody deformation space of maximal decompositions of over finite groups

a rigidity criterion, saying that for a one-ended hyperbolic group , is infinite if and only if has an “interesting” splitting providing an infinite group of Dehn twists,

a particular kind of canonical JSJ decomposition, adapted to the rigidity criterion.
Moreover, we prove that these features are algorithmic: one can compute these invariants, and decide whether the rigidity criterion holds.
The two major algorithmic tools we use are Gerasimov’s algorithm which detects whether
a given hyperbolic group splits over a finite subgroup ([Gerasimov], see also a published version in [DaGr_detecting]),
and a solution to the problem of equations in hyperbolic groups, [DG1].
These algorithms themselves rely on interesting structures. We do not detail them here, instead we refer the interested reader to the indicated bibliography.

Let us now give more details about these structural features and their computation.

#### The Stallings-Dunwoody deformation space.

The Stallings-Dunwoody deformation space of is the set of decompositions of as graphs of groups with finite edge groups and finite or one-ended vertex groups. Existence of such a decomposition for a finitely presented group is Dunwoody’s original accessibility, and Stallings’ theorem shows its relation with the number of ends of .

Gerasimov’s algorithm [Gerasimov, DaGr_detecting] allows one to compute some decomposition in this deformation space. In absence of torsion, the uniqueness property of the Grushko decomposition up to isomorphism reduces immediately the isomorphism problem to the case of freely indecomposable groups.

In presence of torsion, there is no such nice uniqueness statement. Instead, we use the fact that the action of on the Stallings-Dunwoody deformation space is cocompact, and that two reduced trees in this deformation space are connected by slide moves. Starting from a particular decomposition in this deformation space given by Gerasimov’s algorithm, we are able to compute the vertices of the quotient of this deformation space by , i. e. the finite set of all isomorphism classes of reduced Stallings-Dunwoody decompositions. Once this is done, the isomorphism problem for several-ended hyperbolic groups reduces to the isomorphism problem for one-ended hyperbolic groups with a peripheral structure consisting of finite groups. This is done in Section LABEL:sec_ends.

#### The rigidity criterion.

To introduce our version of the rigidity criterion, first consider the torsion-free case. If is a one-ended torsion-free hyperbolic group with infinite, the Bestvina-Paulin argument shows that has a small action on an -tree [Be_degenerations, Pau_topologie]. Then by Rips theory, one can construct a non-trivial splitting (or an HNN extension , but let’s ignore this case in this introduction) over a maximal cyclic subgroup of [Sela_isomorphism, Theorem 9.1]. Conversely, a splitting , any in the centre of defines a Dehn twist as the identity on and the conjugation by on . If is a maximal cyclic subgroup of , then and have finite centre which guarantees that has infinite order in .

In presence of torsion, the class of virtually cyclic groups naturally generalises that of cyclic groups. However, because the infinite dihedral group has trivial centre, there is no non-trivial Dehn twist (in the sense we just discussed) arising from an amalgamated free product , even though is indeed virtually cyclic. With this example in mind, a good dichotomy over virtually cyclic groups is to distinguish whether their centre is infinite or finite. We call -groups the virtually cyclic groups with infinite centre, and -subgroups the -subgroups maximal for inclusion. Only splittings over -groups provide Dehn twists that can be of infinite order (see [MNS_downunder] where this difficulty was already spotted), and the fact that they are of infinite order is guaranteed for splittings over -subgroups. An important observation we make, is that a rigidity criterion remains true in presence of torsion:

###### Proposition 6.

(Rigidity criterion, see Proposition LABEL:prop_alt)

Let be a one-ended hyperbolic group. Then the following assertions are equivalent: {enumerate*}

does not split non-trivially over a -subgroup,

is finite,

There exists such that, modulo inner automorphisms, there are only finitely many endomorphisms of injective on the ball of radius .

For all hyperbolic group , there exists such that, modulo inner automorphisms of , there are only finitely many morphisms injective on the ball of radius of .

See [Lev_automorphisms, Theorem 1.4] for a similar statement. The first part of this equivalence is proved by observing that a splitting over -subgroups provides Dehn twists making infinite. Assuming the negation of the fourth assertion, the Bestvina-Paulin argument produces an action on an -tree, whose careful analysis shows the existence of a splitting.

#### Isomorphism problem and recognition of rigid groups.

An important idea of Sela is to use, in conjunction to such a rigidity criterion, a solution to the problem of equations to get a finite list of morphisms containing a representative of all monomorphisms [Sela_isomorphism]. This approach was simplified in [DaGr_isomorphism], thanks to the use of rational constraints in systems of equations. In [DG1], we developed a solution to the problem of equations with (quasi-isometrically embedded) rational constraints in hyperbolic groups with torsion. We follow the same approach, using this solution to the problem of equations together with our rigidity criterion mentioned above.

Let us describe this approach. Morphisms can be encoded by solutions in of a system of equations corresponding to the presentation of ; injectivity of on the ball of radius can be encoded by inequations; and roughly speaking, rational constraints can be used to ensure that two morphisms do not coincide modulo inner automorphisms of . Thus, given , one can produce a system of equations with rational constraints saying that is a morphism injective on the ball of radius , distinct from modulo inner automorphisms of . Now if Assertion 4 of the rigidity criterion holds, one can enumerate all morphisms , and one will eventually find a finite family such that the corresponding system has no solutions. This attests that every monomorphism is post-conjugate to some .

This argument has two important consequences. First, this allows to recognise whether a one-ended hyperbolic group is rigid (i. e. satisfies Proposition 6) or not. Indeed, if is rigid, one can apply the argument above with , thus attesting that Assertion 3 of Proposition 6 holds. If is not rigid, the fact that Assertion 1 fails can be attested by producing a splitting of . Second, if both and are rigid, one can compute two finite list of morphisms and containing a representative of all monomorphisms up to inner automorphisms. Then one solves the isomorphism problem between and by checking whether there exists such that and are inner automorphisms. This is the content of Section LABEL:subsubsec_resolving.

#### JSJ decompositions.

Thanks to the decidability of the rigidity criterion, one can decide if has a splitting over a -subgroup. Then, using a relative version of the rigidity criterion (Proposition LABEL:prop_alt), one can decide if the vertex groups split over -subgroups relative to the incident edge groups. Iterating this procedure, one can compute a maximal splitting over -subgroups of .

However, such a splitting is not unique up to automorphisms, and we cannot use it to reduce the isomorphism problem to the case of rigid hyperbolic groups (even rigid relative to a peripheral structure). This is why we need a particular kind of JSJ decomposition, encoding splittings over -subgroups.

For any class of subgroups of , invariant under conjugation and stable under taking subgroups, one can define JSJ decompositions over [GL3a]. In general, this defines a deformation space, but in the cases we consider, this deformation space contains a preferred canonical (i. e. -invariant) decomposition, so we speak about the JSJ decomposition over . In a JSJ decomposition, one distinguishes between rigid and flexible vertex groups, according to whether they are elliptic in all splittings of over subgroups in .

We will discuss three possibilities for : the class of virtually cyclic groups and their subgroups, the class of virtually cyclic groups with infinite centre and their subgroups, and the class of maximal -subgroups.

The JSJ decomposition over virtually cyclic groups is now classical. It coincides with Bowditch’s decomposition [Bo_cut] and has been widely studied [DuSa_JSJ, FuPa_JSJ], but we still do not know whether it is computable. Its edge groups are virtually cyclic and its flexible subgroups are hanging bounded Fuchsian groups, i. e. finite extensions of fundamental groups of hyperbolic -orbifolds, possibly with mirrors.

Over the class of -subgroups, the flexible subgroups of the JSJ decomposition are finite extensions of fundamental groups of hyperbolic -orbifolds without mirrors, and edge groups are -groups. Maybe surprisingly, this decomposition can be non-trivial for the fundamental group of a closed orbifold with mirrors. Indeed, we prove that the -JSJ decomposition of an orbifold with mirrors is the splitting over the boundary of a regular neighbourhood of the union of mirrors. The reason is that splittings over -subgroups of are dual to simple closed curves which don’t intersect the singular locus. One can state an interesting corollary of this observation.

aut_hyp_aux \Exportthm\Exportsection\Exportsubsection\Exportsubsubsection \closeexport\Importaut_hyp_sauve

###### Corollary 1.1 (see also [Fujiwara_outer]).

Let be a one-ended hyperbolic group, possibly with torsion (for instance, the fundamental group of a closed orbifold with mirrors ).

Then there is a finite index subgroup of which is an extension

where is virtually abelian, and is a finite index subgroup of the pure extended mapping class group of a surface with boundary.

aut_hyp_aux

Unfortunately, we are not able to compute algorithmically the JSJ decomposition over , so we consider (a variant of) the JSJ decomposition over the class . This decomposition is different from the more usual ones, and it should play a helpful role in order to extend Sela’s program of elementary equivalence among hyperbolic groups to the case of hyperbolic groups with torsion.

As is not stable under taking subgroups, the JSJ decomposition over -subgroups does not fit into the definition of JSJ decompositions from [GL3a], but can nevertheless be defined (see Section LABEL:sec_TZmax).

Its rigid vertex groups are those with no -splitting relative to incident edge groups. According to the rigidity criterion, they are those with finite outer automorphism group relative to incident edge groups. Its flexible vertices are orbisockets. These are finite extensions of fundamental groups of -orbifolds without mirrors, whose boundary subgroups are amalgamated to larger -groups. A typical example consists in adding a root to a boundary component as follows: the orbisocket group is where is the fundamental group of a surface with boundary, and is the fundamental group of a boundary component.

A key step in our proof consists in computing this JSJ decomposition over -subgroups. To do so, one first computes a maximal decomposition of over -subgroups as explained above. In such a decomposition, an orbisocket is cut into pieces, called basic orbisockets. One reconstructs the JSJ decomposition by first recognising the vertex groups of which are basic orbisockets, and by gluing together the pieces that match. This recognition is not immediate, even in the torsion free case [DaGr_isomorphism]. The situation here is even more delicate, and will occupy a significant part of the study.

Once the JSJ decompositions of one-ended hyperbolic groups have been computed,
the isomorphism problem reduces to the isomorphism problem for the vertex groups (with marked peripheral structures).
A relative version of the isomorphism problem for rigid groups (relative to marked peripheral structure)
allows to do so for rigid vertex groups,
and the isomorphism problem for orbisockets
is easy once the basic orbisockets it is made of are identified.

Let us comment the structure of the paper. We decided to include a rather extended toolbox, in which we recall classical, but sometimes subtle, material, including elements of Bass-Serre theory, and isomorphisms of graphs of groups. Section LABEL:sec_rigid is devoted to the rigidity criterion, and its application to the isomorphism problem for rigid hyperbolic groups. In Section LABEL:sec_JSJ, we introduce the definition and properties of the JSJ decompositions over and -subgroups, and we introduce orbisockets as flexible vertices of the JSJ decomposition. Sections LABEL:sec_orbi and LABEL:sec_1end are mainly devoted to the computation of this JSJ decomposition. The main part of section LABEL:sec_orbi is devoted to the recognition of basic orbisockets, from which follows a solution of the isomorphism problem for orbisockets with their marked peripheral structure. In section LABEL:sec_1end, we compute the JSJ decomposition by gluing together basic orbisockets of some non-canonical maximal decomposition, and we conclude our solution of the isomorphism problem for one-ended hyperbolic groups, Section LABEL:sec_ends is devoted to hyperbolic groups with several ends, and to the computation of the Stallings-Dunwoody deformation space, and finishes the solution to the isomorphism problem for all hyperbolic groups. Finally, in Section LABEL:sec_Wh, we prove a relative version of the isomorphism problem, and we deduce a solution of Whitehead problems.

## 2 Tool box

### 2.1 Actions of finitely generated groups on finite sets

Several times, we will use orbit decidability of group actions on finite sets. The following lemma is rather elementary, but we need a somewhat general statement.

###### Lemma 2.1.

(Orbit decidability in finite sets) Let be a group acting on a finite set , and a set with a surjection on . Assume that the following is known: {itemize*}

a finite generating set of ,

an algorithm deciding whether two given elements of have same image in .

an algorithm that, given and , computes an element with , Then given two elements in , one can decide whether and are in the same orbit under the action of . If they are in the same orbit, one can compute an element of (as a word on ) sending to .

Moreover, given , one can compute a generating set of the stabiliser of .

Let us emphasize that the entire set , and even its cardinality, is not assumed to be known, that might be infinite, and that does not act on .

###### Proof.

Consider and . Let be the set of images of under elements of of length at most .

By hypothesis, one can compute from representatives in of . One can also check whether and check whether . Since is finite, for some , so since is invariant under the generators of . One can therefore check whether lies in . In this case, one easily finds a word of length sending to .

Let’s compute generators for the stabiliser of . The argument above allows one to compute the Schreier graph of the action of on : its vertex set is and are joined by a directed edge labelled by if . One can obtain a set of generators of by considering the words labelled by a generating set of (see also [MKS_combinatorial, Theorem 2.7, p.89]). ∎

### 2.2 Extensions

Let be a fixed group, and , some extensions of by . We say that an isomorphism lifts if there exists making the following diagram commute: