The Status of Polycyclic GroupBased Cryptography: A Survey and Open Problems
Abstract.
Polycyclic groups are natural generalizations of cyclic groups but with more complicated algorithmic properties. They are finitely presented and the word, conjugacy, and isomorphism decision problems are all solvable in these groups. Moreover, the nonvirtually nilpotent ones exhibit an exponential growth rate. These properties make them suitable for use in groupbased cryptography, which was proposed in 2004 by Eick and Kahrobaei [10].
Since then, many cryptosystems have been created that employ polycyclic groups. These include key exchanges such as noncommutative ElGamal, authentication schemes based on the twisted conjugacy problem, and secret sharing via the word problem. In response, heuristic and deterministic methods of cryptanalysis have been developed, including the lengthbased and linear decomposition attacks. Despite these efforts, there are classes of infinite polycyclic groups that remain suitable for cryptography.
The analysis of algorithms for search and decision problems in polycyclic groups has also been developed. In addition to results for the aforementioned problems we present those concerning polycyclic representations, group morphisms, and orbit decidability. Though much progress has been made, many algorithmic and complexity problems remain unsolved; we conclude with a number of them. Of particular interest is to show that cryptosystems using infinite polycyclic groups are resistant to cryptanalysis on a quantum computer.
result[Table][List of Results]
Contents
 1 Introduction
 2 Algorithmic Problems in Polycyclic Groups

3 Cryptosystems
 3.1 The AnshelAnshelGoldfeld KeyExchange Protocol
 3.2 KoLee Key Exchange Protocol
 3.3 NonCommutative ElGamal KeyExchange
 3.4 NonCommutative Digital Signature
 3.5 A Key Exchange Using the Subgroup Membership Search Problem
 3.6 An Authentication Scheme Based on the Twisted Conjugacy Problem
 3.7 Authentication Schemes Based on Semigroup Actions
 3.8 Secret Sharing Schemes Based on the Word Problem
 4 Cryptanalysis and Attacks
 5 Conclusion
1. Introduction
In cryptography, many of the most common key exchange protocols, including RSA and DiffieHellman, rely upon hardness assumptions related to integer factorization and discrete logarithms for their security. While there are no known efficient algorithms for performing the above operations on conventional computers, Peter Shor devised a quantum algorithm [39] that solves both of these problems in polynomial time. This has motivated the search for alternative methods for constructing cryptosystems. One such methodology is noncommutative cryptography, which unlike the aforementioned conventional systems does not operate over the integers. Instead, noncommutative cryptographic systems are built upon groups and other algebraic structures whose underlying operations are noncommutative.
In 1999, Anshel, Anshel, and Goldfeld [1] and Ko, Lee, et al. [25] introduced key exchange protocols whose security is based in part on the conjugacy search problem: for a group , given that are conjugate, find an in such that . Though braid groups were the suggested platform for both protocols, other classes of groups can be employed. In general, groups suitable for use in noncommutative cryptography must be wellknown and possess the following properties: a solvable word problem, a computationally difficult grouptheoretic problem, a “fast” word growth rate, and the namesake noncommutativity [33].
In 2004, Eick and Kahrobaei [10] investigated the algorithmic properties of polycyclic groups. In particular, they explored how the time complexity of the word and conjugacy problems varied with respect to a group’s Hirsch length. Their experiments showed that the while the time complexity of the conjugacy problem grew exponentially with increased Hirsch length, the word problem remained efficiently solvable. These results suggested the suitability of polycyclic groups for use in cryptography, and stimulated research into cryptosystems based on these groups and their underlying algorithmic problems.
In this paper, we survey the development of groupbased cryptography over polycyclic and metabelian groups. In section 2 we discuss the algorithmic properties of polycyclic groups. Polycyclic groups and their intrinsic presentations are defined, as well as several other representations. A number of grouptheoretic decision problems are introduced, including the word, conjugacy, and isomorphism decision problems. Note that in every polycyclic group, the three aforementioned problems are solvable. Moreover, the word problem can be solved efficiently in most cases by using a collection algorithm.
In section 3 we describe a number of cryptosystems that have been built around these groups. These include additional key exchanges along with schemes for secret sharing, authentication, and digital signatures. This variety of cryptosystems evinces the flexibility and utility of polycyclic groups in noncommutative cryptography.
As new cryptosystems are created, so too are their dual in the form of cryptanalyses and attacks. In section 4 we discuss the lengthbased attack, a heuristic technique that was the first to break the AAG protocol over braid groups. Other attacks exploit the linear representation that all polycyclic groups admit. Some, such as the fieldbased attack, are specific to a subclass of polycyclic groups. A more general approach is the linear decomposition attack, but its feasibility is dependent upon the size of a group’s representation.
We conclude the paper with the current status of polycyclic groups cryptography. We also include a list of open problems, which we hope will guide researchers who wish to work in this exciting field.
2. Algorithmic Problems in Polycyclic Groups
The nature of polycyclic groups enables them to be represented in several ways. These approaches give rise to complementary algorithms for solving search and decisions problems, with varying degrees of computational complexity. Due to this flexibility, we begin our study of the algorithmic problems in polycyclic groups by examining these representations.
2.1. Representations of Polycyclic Groups
2.1.1. Polycyclic Sequences and Hirsch Length
A group is said to be polycyclic if it has a subnormal series such that the quotient groups are cyclic. This series is called a polycyclic series. The Hirsch length of a polycyclic group is the number of infinite groups in its polycyclic series. Though a polycyclic group can have more than one polycyclic series, as a consequence of the Schreier Refinement Theorem, its Hirsch length is independent of the choice of series.
2.1.2. Polycyclic Presentations
Every polycyclic group can be described by a polycyclic presentation:
where are words in the generators and is the set of indices such that is finite.
This special type of finite presentation reveals the polycyclic structure of the underlying group, see [20, Chapter 10] for details. Unlike general finite presentations, a polycyclic presentation enables the word problem to be solved using an algorithm called collection. The collection algorithm is generally effective in practical applications, but its precise computational complexity remains unknown. For finite groups, collection from the left was shown to be polynomial by LeedhamGreen and Soicher [27]. For infinite groups, the complexity of the collection algorithm and a modified version were analyzed by Gebhardt [16]. The resultant worstcase bound is in terms of the absolute values of all exponents occurring during the collection process, rather than the exponents of the input word. Thus a global complexity analysis of the collection algorithm remains elusive.
2.1.3. Polycyclic Presentations with a Malcev Basis
It has been shown by Assmann and Linton [2] that the efficacy of the collection algorithm can be improved significantly by exploiting the Malcev structure of the underlying group. This approach determines a large nilpotent normal subgroup of the given group and then exploits the Malcev correspondence for the normal subgroup. There is no known complexity analysis for this methodology.
2.1.4. Polycyclic Presentations with Multiplication Polynomials
Du Sautoy [8] proved that every polycyclic group has a normal subgroup of finite index such that multiplication in this subgroup can be achieved by evaluating certain multiplication polynomials. This extends the wellknown result by Hall [19] for torsionfree nilpotent polycyclic groups. If such multiplication polynomials are available the performance of collection in the considered group improves significantly. Additionally, it provides a basis for the complexity analysis of multiplication in polycyclic groups; it must be noted however that the index of the normal subgroup can be arbitrarily large.
2.1.5. Matrix Groups
It is wellknown that every polycyclic group can be embedded into for some . For groups that are additionally torsionfree and nilpotent, a matrix representation can be computed. The algorithm of Lo and Ostheimer [28] can be applied to a polycyclic presentation, while for multiplication polynomials the technique by Nickel [35] can be utilized. Multiplication of group elements in their matrix form is polynomial in the dimension of the representation.
2.2. Growth Rate
Let be a finitely generated group. The growth rate of a group is specified by its growth function defined as , where is the length of as a word in the generators of . As words are used as keys in groupbased cryptography, there is a natural relationship between the growth rate of a group and the key space, the set of all possible keys. A fast growth rate engenders a large key space, making an exhaustive search of this space intractable.
2.3. Decision Problems
In 1911, Max Dehn introduced [7] three decision problems on finitely presented groups  the word problem, the conjugacy problem, and the isomorphism problem. In the definitions below, let be a finitely presented group:

Word Decision Problem  For any , determine if , the identity element of .

Single Conjugacy Decision Problem  Determine for any if is conjugate to (denoted ).

Isomorphism Decision Problem  Given groups and with respective finite presentations and , determine if is isomorphic to .
For polycyclic groups all three of the above problems are decidable. The conjugacy decision problem for polycyclic groups is decidable by the results of Remeslennikov [36] and Formanek [13]. That the word problem is decidable can be observed from its formulation as a special case of the conjugacy decision problem (where ), or by observing that every word has a unique normal form induced by a polycyclic presentation. The isomorphism decision problem for polycyclic groups is solvable by a result of Segal [38].
An additional decision problem called the subgroup membership decision problem (alternatively the generalized word decision problem) asks for any and subgroup , determine if . Malcev in [29] showed that this problem is indeed solvable for polycyclic groups.
2.4. The Conjugacy Search Problem and its Variations
Once the solvability of a grouptheoretic decision problem is affirmed, the subsequent task is to produce elements (or morphisms, etc.) that are solutions to particular instances of it. The seminal protocols of noncommutative cryptography, KoLee and AAG, are based in part on the conjugacy search problem (CSP). Their example spurred the development of many other protocols whose security is based on some variant of the CSP. In this section we explore these variations and the methods designed to solve them.
2.4.1. Conjugacy Search Problem
Let be a group and elements of it, with . The problem of finding a such that for all , is called the (single) conjugacy search problem for and the multiple conjugacy search problem for . In polycyclic groups, the multiple conjugacy search problem for elements reduces to independent solutions of single conjugacy search [10]. We will therefore speak only of the conjugacy search problem without signifying arity.
For any finitely presented group (polycyclic groups included) the conjugacy search problem can be solved exhaustively by recursively enumerating the conjugates of the element in question [40]. There are other approaches to solving the conjugacy search problem, many of which can solve it efficiently. However, the applicability of these methods and their relative efficiency is contingent upon addition restrictions on the group’s properties, as well as the manner is which the polycyclic group is specified.
2.4.2. CSP Using Polycyclic Presentations
For infinite polycyclic groups the algorithm proposed by Eick and Ostheimer [11] is applicable. This algorithm uses a variety of ideas: it exploits finite orbit and stabilizer computations, calculations in number fields, and linear methods for polycyclic groups. The algorithm has been implemented and seems to be efficient for groups of small Hirsch length. An analysis of the algorithm’s complexity is hindered by there being no bound on the length of the finite orbits that may occur in the computation.
The restriction of the applicability of the above algorithm to groups of small Hirsch length is supported by the experimental evidence provided by Eick and Kahrobaei in [10]. They compared the performance of the EickOstheimer algorithm for the CSP against the collection algorithm for polycyclic groups of the form , where and are respectively the maximal order and group of units of an algebraic number field . In the table below, the column is the Hirsch length of the group , with the collection and conjugation entries representing the average running time over 100 trials using random words (respectively, random conjugate pairs) from :
H(G)  Collection  Conjugation 

2  0.00 sec  9.96 sec 
6  0.01 sec  10.16 sec 
14  0.05 sec  hr 
These results suggest that while collection remains efficient as the Hirsch length increases, the EickOstheimer algorithm becomes impractical. Presently there are no known algorithms for infinite polycyclic groups of high Hirsch length. Such groups remain suitable for use as platform groups.
2.4.3. CSP Using Multiplication Polynomials
Suppose that instead is given by a polycyclic presentation with multiplication polynomials. Let be the polycyclic generating set of the presentation and consider a generic element of . is a solution to the multiple conjugacy search problem if and only if for . If and , with denoting the multiplication polynomials for , then if and only if
If are given as explicit polynomials over an extension field of and are integer vectors, then the CSP is equivalent to determining an integer solution for a set of polynomials in indeterminates. Thus the CSP can also be considered from the perspective of algebraic geometry.
2.4.4. Power Conjugacy Search Problem
The key exchange presented in Section 3.3.2 makes use of the power conjugacy search problem, where if it is known for some and that for some , the task is to find one such and . Note that for this reduces to the standard CSP, whereas if this reduces to the power search problem.
Just as the conjugacy search problem is solvable by enumeration, so is the power conjugacy search variant, but no efficient algorithm is known.
2.4.5. Twisted Conjugacy Search Problem
Twisted conjugacy arises in Nielsen theory, where the number of twisted conjugacy classes is related to number of fixed points of a mapping. The twisted conjugacy search problems is to find, given a group and an endomorphism , an element such that , provided that at least one such exists.
The standard CSP can be seen as a special case of the twisted version where , the identity automorphism. The protocol by Shpilrain and Ushakov in Section 3.6 uses the double twisted conjugacy variant, in which the above definitions is modified to include an additional endomorphism and the task is to then find an element such that .
The twisted conjugacy decision problem was proven to be decidable by Roman’kov [37]. Both the single and doubly twisted conjugacy search problems are solvable by the same method of enumeration as in the case of the standard conjugacy search problem. However, no efficient algorithm is known.
2.5. Properties of Automorphism Groups
The automorphism group and its subgroups have been extensively studied for polycyclic groups . Like polycyclic groups themselves, is finitely presented [3], and the outer automorphism group is isomorphic to a linear group [45].
A decision problem related to is the orbit decision problem. Given elements and a subset , determine if there exists such that . Note that if this problem reduces to the standard conjugacy decision problem. When is polycyclic all cyclic subgroups are orbit decidable [5].
2.6. Quantum Algorithms
As mentioned in the introduction, the introduction noncommutative cryptography was spurred by the publication of Shor’s algorithm. The algorithm enables a sufficiently sized quantum computer to perform integer factorization and compute discrete logs in polynomial time, as opposed to in exponential time on a conventional computer.
From a grouptheoretic perspective, Shor’s algorithm can be seen as solving the hidden subgroup problem in finite cyclic groups. A subgroup is considered hidden by a function from to a set if it constant over all cosets of . A 2003 paper by [4] by Batty, et al. explores this and other applications of quantum algorithms to group theory, including an algorithm by Watrous that determines the order of a finite solvable group. Bonanome showed [6] that a modified version of Grover’s algorithm can solve the automorphism and conjugacy decision problems in finite groups, as well as determine fixed points. The algorithm by Ivanyos, et al [22] solves the hidden subgroup problem for finite nilpotent groups of class 2. There are also partial results to solving the power conjugacy problem [12].
Despite these developments in the use quantum algorithms for finite groups, there are no known quantum algorithms that are applicable to infinite groups.
3. Cryptosystems
For the systems described below, the chosen platform group should be suitable for cryptography as delineated in the introduction. Let be finitely presented and nonabelian. Group operations (products, inverses) and solving the word problem must be efficient. Additional criteria for each protocol or scheme are stated in their respective descriptions. Note that the precise definitions of each algorithmic search or decision problem can be found in Section 2.
3.1. The AnshelAnshelGoldfeld KeyExchange Protocol
In their 1999 paper [1], Anshel, Anshel, and Goldfeld introduced the commutator key exchange protocol, which is also referred to as AAG key exchange or Arithmetica. The groupbased version of the key exchange described below is in the style of [31]. Prior to the key exchange, the protocol parameters , with , are chosen and made public:

Alice chooses a set , with Bob choosing , where are words of length in . Note that and both generate subgroups of . These sets are then exchanged publicly with each other.

Alice constructs her private key as , with and . Similarly, Bob computes as his private key , with and .

Alice then computes for and sends this collection to Bob, while Bob computes and sends Alice for .

Alice and Bob can now compute a shared key , which is the commutator of and , denoted . Alice computes (using only the which correspond to some of her private key):
Analogously, Bob computes . The shared secret is then .
As noted in [41], the security of AAG is based on both the simultaneous conjugacy search problem and the subgroup membership search problem.
3.2. KoLee Key Exchange Protocol
Originally specified by Ko, Lee, et al. [25] using braid groups, their noncommutative analogue of DiffieHellman key exchange can be generalized to work over other platform groups. Let be a finitely presented group, with such that all elements of and commute.
An element is chosen, and are made public. A shared secret can then be constructed as follows:

Alice chooses a random element and sends to Bob.

Bob chooses a random element and sends to Alice.

The shared key is then , as Alice computes , which is equal to Bob’s computation of as and commute.
The security of KoLee rests upon solving the conjugacy search problem within the subgroups .
3.3. NonCommutative ElGamal KeyExchange
In the 2006 paper by Kahrobaei and Khan [23], the authors proposed two adaptations of the ElGamal asymmetric key encryption algorithm for use in noncommutative groups. Let be finitely generated subgroups such that all elements of and commute. In any exchange, the triple is made public.
3.3.1. NonCommutative Key Exchange Using Conjugacy Search

Bob chooses as his private key, a random element , and publishes as his public key the tuple , with .

To create a shared secret , Alice chooses and a . Using Bob’s public key, she publishes , with and .

To recover , Bob first computes , which, as elements of and commute, yields
Bob can then calculate .
The security of this scheme relies upon the conjugacy search problem in .
3.3.2. NonCommutative Key Exchange Using Power Conjugacy Search
By imposing the additional requirement that the conjugacy search problem is efficiently solvable in , we can now describe a variation of the previous protocol:

Bob chooses and as his private key, as well as a random element , and publishes as his public key , with and . Note that .

Alice chooses a shared secret , along with and , and publishes , with and .

To recover , Bob first computes , which, as elements of and commute, yields
Knowing that , Bob can then solve the conjugacy search problem to obtain the shared secret .
The security of this scheme rests upon the power conjugacy search problem in .
3.4. NonCommutative Digital Signature
The following digital signature scheme was proposed in a paper by Kahrobaei and Koupparis [24]. The platform group must be infinite. The scheme uses two functions: , which encodes elements of the group as binary strings; and , a collisionresistant hash function. Using these functions (which are made public along with ), a message can be signed and verified as follows:

Key Generation: The signer first chooses an element , whose centralizer, the set of elements that commute with , contains and powers of exclusively. The private key consists of and , where is chosen to be highly composite. The public key is then published.

Signing Algorithm: To sign a message , the signer chooses a random element and a random factorization of , and computes the following (with denoting concatenation):
The signature and the message are then send to the message recipient.

Verification: To verify, the recipient computes , and accepts the message as authentic if and only if the following equality holds:
The security of the signature scheme is based on the collision resistance of the hash function, the conjugacy search problem in , and the DiffieHellman assumption. Moreover, Alice must maintain a public list of previously used factors of , and regenerate and after a few uses.
3.5. A Key Exchange Using the Subgroup Membership Search Problem
In [43], Shpilrain and Zapata proposed a public key exchange protocol over relatively free groups. Given a free group of rank and , the quotient group is relatively free if for any endomorphism of , .
The protocol utilizes two types of automorphisms:

Let be the generators of . The Neilsen automorphisms are defined as:

For relatively free groups like , the Nielsen automorphisms form a subgroup of under composition. Elements in this subgroup are called tame automorphisms. In constructing a private key, the protocol uses both tame and nontame automorphisms.
In the key exchange below, let and denote the relatively free groups of rank and , with respective generating sets and
. Moreover, let denote the direct product of instances of the relatively free group of rank . Finally, let denote a word written in the alphabet . The exchange then proceeds as follows:

Alice chooses an automorphism , where , a composition of Nielsen automorphisms and nontame automorphisms which are readily invertible. Alice uses as her private key. For each generator of , Alice computes the word . She then computes , which is the restriction of each to a word in the generators of . The tuple is then published as the public key.

Bob chooses a word in the subgroup of consisting of words of the form . Thus , and . Using the components of the public key, Bob encrypts by replacing each instance of in by . The encrypted tuple is then sent to Alice.

Alice applies (restricted to ) componentwise to to recover , a unique normal form of . This is the shared key.
The security of the protocol is twofold. Decrypting a particular message is equivalent to solving the subgroup membership search problem in the subgroup generated by the public key. To recover the private key, an attacker must recover the automorphism and its inverse from the public image of the generators , restricted to the subgroup . Shpilrain and Zapata claim there is no known method of accomplishing this outside of an exhaustive search of .
The authors suggest free metabelian groups of rank (with ) as platform groups for their protocol. Aside from meeting the standard criteria for platform groups, these groups have the requisite supply of nontame automorphisms and the subgroup membership search problem is known to be superpolynomial in these groups.
3.6. An Authentication Scheme Based on the Twisted Conjugacy Problem
In [42], Shpilrain and Ushakov introduced a noncommutative authentication scheme based on the FiatShamir scheme. The platform group can in fact be a semigroup, provided that an antihomomorphism , i.e., , exists. The endomorphism group of should also be sufficiently large to preclude an exhaustive search. In the simulation of the protocol below, Alice is authenticating herself to Bob:

Alice chooses as her private key. She then chooses and endomorphisms such that . The public key is then published.

The commitment/verification exchange proceeds as follows:

Alice chooses an and computes the commitment , sending it to Bob.

Bob chooses a random bit and sends it to Alice.

Alice replies with if , and otherwise.

Bob verifies the commitment by computing , and accepts if :
If , Bob computes .
If , Bob computes , where

Note that the commitment/verification steps must be performed times to yield a probability of successful forgery less than . The security of the scheme is based on the apparent hardness of the double twisted conjugacy search problem.
3.7. Authentication Schemes Based on Semigroup Actions
Drawing inspiration from the zeroknowledge proof by Feige, Fiat, and Shamir; Grigoriev and Shpilrain [17] introduced two generic protocol schema based upon (semi)group actions and provided several concrete examples.
3.7.1. An Authentication Scheme Based on the Endomorphism Problem
One such instance of their second protocol is based upon the endomorphism problem. While this scheme can be used with a semigroup or some other algebraic structure, the structure must meet several criteria:

An algorithm exists to determine if function over is an endomorphism. If is specified by a presentation this criterion is satisfied by having an efficiently solvable word problem.

An algorithm exists to determine if function over is an automorphism of .

The endomorphism search problem in should be demonstrably NPhard.
As before, in the protocol exchange below Alice is authenticating herself to Bob:

Alice chooses an endomorphism as her private key. Alice then chooses elements such that . The public key is then published.

The commitment/verification exchange proceeds as follows:

Alice chooses an automorphism and computes the commitment , sending it to Bob.

Bob chooses a random bit and sends it to Alice.

Alice replies with if , and otherwise.

Bob verifies the commitment by computing :
If , Bob computes and accepts if and is an automorphism.
If , Bob computes and accepts if and is an endomorphism.

3.7.2. An Authentication Scheme Based on the Group Isomorphism Problem
The following is a new instance of the first protocol, which requires a class of finitely presented groups with the following algorithmic properties:

The class must have an efficiently solvable isomorphism decision problem.

The isomorphism search problem in should be demonstrably NPhard.
The protocol exchange is as follows:

Alice chooses two isomorphic groups and from . Alice then chooses an isomorphism as her private key, and publishes .

The commitment/verification exchange proceeds as follows:

Alice chooses a group and an isomorphism , sending the commitment to Bob.

Bob chooses a random bit and sends it to Alice.

Alice replies with if , and otherwise.

Bob verifies the commitment by computing :
If , Bob accepts if .
If , Bob accepts if .

For both of the above authentication schemes, the commitment/verification steps must be performed multiple times to yield a low probability of successful forgery.
3.8. Secret Sharing Schemes Based on the Word Problem
Habeeb, Kahrobaei, and Shpilrain [18] proposed two secret sharing schemes for groups whose presentations satisfy small cancellation conditions. In a scheme, the threshold is the number of participants that are required to recover the shared secret (created and disseminated by the “dealer”), with the total number of participants.
In both schemes, the dealer wishes to share a bit integer that will be represented as a column vector . Prior to initiating the secret sharing, the dealer chooses groups given by the presentations , where is a common generating set and a unique set of relators for each participant . The generating set is then made public. Note that both schemes require secure communication channels between both the dealer and participants and between the participants themselves. These secure channels can be achieved using any preferred public key exchange protocol.
3.8.1. An threshold Scheme
In this scheme, all participants are required to reproduce the secret :

The dealer sends each participant their unique relator set .

The dealer decomposes into vectors such that .

Each entry of is then encoded as a word , such that if and otherwise. The s are then sent to using an open channel.

For each , participant solves the word problem in and reconstructs .

The participants can then recover by summing over all s. Note that a secure sum protocol can be employed so that the s need not be divulged to the other participants.
3.8.2. A threshold Scheme
In this scheme, participants are required to reproduce the secret . As in Shamir’s secret sharing, must be an element in with prime, and a polynomial of degree must be chosen by the dealer such that . The dealer must also choose bit integers .

The dealer sends each participant their unique relator set .

Each has its bits encoded as words as in the previous scheme.

For each , participant solves the word problem in , yielding .

The participants can then perform polynomial interpolation using the s to recover . The shared secret is then revealed by evaluating . If , Lagrange interpolation can be employed so that the s need not be divulged to the other participants.
The security of these schemes is contingent upon the relators being kept secret.
4. Cryptanalysis and Attacks
In this section we present a number of attacks against groupbased cryptosystems, with an emphasis on those that are applicable to polycyclic groups.
4.1. LengthBased Attack
The lengthbased attack (LBA) is an incomplete, local search that attempts to solve the conjugacy search problem (or its generalized version) by using the length of a word as a heuristic. It was first introduced by Hughes and Tannenbaum [21] as a means to attack the AAG key exchange protocol over braid groups. In [15], Garber, Kaplan, Teicher, Tsaban, and Vishne explored the use of length functions based on the Garside normal form of braid group elements. They demonstrated experimentally that the lengthbased attack in this context could break the AAG protocol, albeit inefficiently.
As the lengthbased attack is an iterative improvement search, it is susceptible to failing at peaks and plateaux in the search space. In [31], Myasnikov and Ushakov identified when these peaks occur and were able to make successive refinements to the algorithm to yield a high success rate.
More recently, the authors of [14] analyzed the LBA against AAG over polycyclic groups. They found that the success rate of the LBA decreased as the Hirsch length of the platform group increased. Their version of the LBA, essentially a local beam search, is presented below:
Note that the , , , and are from the AAG protocol exchange in Section 3.1, while is a candidate conjugator set. The length of a conjugator set is defined as .
4.2. Linear Decomposition Attack
In [32], Miasnikov and Roman’kov introduced the linear decomposition attack. The attack is a general framework for the cryptanalysis of a number of grouptheoretic analogues of DiffieHellman key exchange. For a protocol to be susceptible to the attack its platform groups must admit a linear representation. Moreover, the algorithmic security assumption of the protocol must be equivalent to commutative linear transformations. Note that the AAG protocol is not susceptible to this attack.
Given the linear structure and subsets and , the attack first computes a basis for the span of all vectors of the form , with and . This can be done in polynomial time with respect to the dimension of and the sizes of and . This calculation can be performed offline if the platform group for a particular protocol is fixed. The public group elements transmitted during the key exchange can then be decomposed using this basis to reconstruct the shared secret without discovering the private information of each party, negating the need for an attacker to solve the underlying security problem.
The attack requires the platform group to be specified by either its linear representation (as a vector space or an associative algebra) or by a presentation coupled with a faithful embedding into . Moreover, the linear space into which the group is embedded must be of sufficiently small dimension to make the attack tractable. While the dimension of the smallest linear embeddings of finite groups and some classes of infinite groups such as torsionfree nilpotent and polycyclicbyfinite are known, the authors concede that no such bounds are known for other linear groups, including general polycyclic groups and metabelian groups.
4.3. FieldBased Attack
Kotov and Ushakov [26] investigated the security of the AAG keyexchange protocol used with certain polycyclic groups of the form , where is the maximal order and is the unit group generated by an irreducible polynomial in the algebraic number field . In the semidirect product, acts on by right multiplication. These groups were the original polycyclic platform groups suggested by Eick and Kahrobaei in [10]. In [14], Garber, Kahrobaei, and Lam showed that such groups were resistant to the lengthbased attack, with the attack’s success decreasing as the Hirsch length of the group increased.
Contrary to these results, the fieldbased attack devised by the authors is able to recover the shared key regardless of the group’s Hirsch length. Using a deterministic, polynomial time algorithm, the key is recovered by solving a linear system of conjugacy equations over the field . If the group is specified as a semidirect product and is given in matrix form, the attack can be directly applied. However, if is given by a polycyclic presentation, the authors construct a linear representation from the presentation prior to recovering the shared key.
While the fieldbased attack is successful in these particular groups, the authors concede that their attack does not preclude other polycyclic groups from consideration for the AAG protocol. We claim that there are other classes of polycyclic groups that are resistant to such an attack. Such platform groups would be specified by their polycyclic presentations and have matrix representations that are not readily computable.
4.4. Quotient Attack
In attempting to recover the shared secret from the public information of the AAG protocol, the lengthbased attack (LBA) operates as if the platform group is a free group. The success of the LBA on nonfree groups motivated Miasnikov and Ushakov in [34] to investigate the asymptotic properties of the given platform groups. Ultimately they determined that the LBA is successful for groups in which a random choice of elements is very likely to generate a free subgroup of .
These investigations led to a new form of attack for the AAG key exchange protocol and others that use some variation of the membership or conjugacy search problems. Dubbed the quotient attack, the algorithms solve the search problems in a quotient group . If possesses the exponentiallygeneric free basis property the solution in the quotient will yield one in the original group. The time complexity of the attack is contingent upon the particular class of platform groups. For pure braid groups the authors prove that the complexity is .
As polycyclic groups do not possess the free basis property nor any free subgroups, this attack is not applicable.
4.5. Linear Centralizer Attack
Tsaban [44] devised the linear centralizer attack against AAG over the original braid group platform. The attack exploits a faithful linear representation of a braid group . Using this representation, the algorithm computes a basis for the double centralizer of the public subsets of the AAG protocol (which are contained in their respective double centralizers). This process produces one half of the shared key, after which random elements are tested to find an inverse that yields the other half. The algorithm runs in expected polynomial time with respect to , but is impractical for even modest values of .
The applicability of the linear centralizer attack to other platform groups is limited to those whose faithful representations are known and whose linear representations are sufficiently small. As mentioned previously with respect to the linear decomposition attack, these aspects of polycyclic groups are currently unknown.
5. Conclusion
In this paper we have presented a survey of over ten years of research related to polycyclic groupbased cryptography. We began with a study of the algorithmic properties of polycyclic groups. Polycyclic groups admit a number of representations, including polycyclic presentations, multiplication polynomials, and as matrices. In addition to the decidability of the classic decision problems of word, conjugacy, and isomorphism, the twisted conjugacy and orbit problem are also decidable. Moreover, the conjugacy decision problem for the automorphism group of a polycyclic group is decidable.
We have seen that there are a variety of key exchanges, digital signature systems, and secret sharing schemes for which a polycyclic group is an appropriate choice of platform group. These schemes use several different computational problems in polycyclic groups as listed in the paper, which are beyond use of conjugacy search problem.
While there has been considerable research activity concerning polycyclic groups and their attendant cryptosystems over the last decade, many computational complexity and algorithmic questions remain unanswered. We have collected these outstanding problems below, with the hope of stimulating interest in their solutions:

What is the complexity of the isomorphism search problem in polycyclic groups?

What is the complexity of the twisted search conjugacy problem in polycyclic groups?

What is the complexity of the power conjugacy problem in polycyclic groups?

What is the complexity of the geodesic length problem in polycyclic groups?

What is the complexity of the root problem in polycyclic groups?

What is the complexity of finding matrix representation of polycyclic groups?

What is the complexity of the conjugacy problem in the automorphism of polycyclic groups?

What is the complexity of the search endomorphism (automorphism) problem in polycyclic groups?

What is the complexity of the homomorphism problem in polycyclic groups?

Are polycyclic groupbased cryptosystems resistant to quantum algorithms?

What is the complexity of the subgroup membership search problem in polycyclic groups?
Acknowledgements
We would like to thank Bettina Eick for her contributions regarding polycyclic groups and their algorithmic properties. Delaram Kahrobaei is partially supported by a PSCCUNY grant from the CUNY Research Foundation, the City Tech Foundation, and ONR (Office of Naval Research) grant N000141512164. Delaram Kahrobaei has also partially supported by an NSF travel grant CCF1564968 to IHP in Paris.
References
 [1] I. Anshel, M. Anshel, and D. Goldfeld. An algebraic method for publickey cryptography. Math. Res. Let., 6:287–291, 1999.
 [2] B. Assmann and S. Linton. Using the Mal’cev correspondence for collection in polycyclic groups. J. Algebra, 316(2):828–848, 2007.
 [3] L. Auslander. The automorphism group of a polycyclic group. Annals of Mathematics, pages 314–322, 1969.
 [4] M. Batty, S. Rees, S. Braunstein, and A. Duncan. Quantum algorithms in group theory. Technical report, 2003.
 [5] O. Bogopolski, A. Martino, and E. Ventura. Orbit decidability and the conjugacy problem for some extensions of groups. Transactions of the American Mathematical Society, 362(4):2003–2036, 2010.
 [6] M. Bonanome. Quantum Algorithms in Combinatorial Group Theory. PhD dissertation, City University of New York, 2007.
 [7] M. Dehn. Über unendliche diskontinuierliche gruppen. Mathematische Annalen, 71(1):116–144, 1911.
 [8] M. du Sautoy. Polycyclic groups, analytic groups and algebraic groups. Proc. London Math. Soc. (3), 85(1):62–92, 2002.
 [9] B. Eick. When is the automorphism group of a virtually polycyclic group virtually polycyclic? Glasgow Mathematical Journal, 45(03):527–533, 2003.
 [10] B. Eick and D. Kahrobaei. Polycyclic groups: a new platform for cryptography, preprint arxiv: math.gr/0411077. Technical report, 2004.
 [11] B. Eick and G. Ostheimer. On the orbitstabilizer problem for integral matrix actions of polycyclic groups. Math. Comp., 72(243):1511–1529 (electronic), 2003.
 [12] A. Fesenko. Vulnerability of cryptographic primitives based on the power conjugacy search problem in quantum computing. Cybernetics and Systems Analysis, 50(5):815–816, 2014.
 [13] E. Formanek. Conjugate separability in polycyclic groups. Journal of Algebra, 42(1):1–10, 1976.
 [14] D. Garber, D. Kahrobaei, and H. T. Lam. Lengthbased attack for polycyclic groups. Journal of Mathematical Cryptology, De Gruyter, pages 33–44, 2015.
 [15] D. Garber, S. Kaplan, M. Teicher, B. Tsaban, and U. Vishne. Lengthbased conjugacy search in the braid group. Contemp. Math. 418, pages 75–87, 2006.
 [16] V. Gebhardt. Efficient collection in infinite polycyclic groups. J. Symbolic Comput., 34(3):213–228, 2002.
 [17] D. Grigoriev and V. Shpilrain. Zeroknowledge authentication schemes from actions on graphs, groups, or rings. Ann. Pure Appl. Logic, 162:194–200, 2010.
 [18] M. Habeeb, D. Kahrobaei, and V. Shpilrain. A secret sharing scheme based on group presentations and the word problem,. Contemp. Math., Amer. Math. Soc., 582:143–150, 2012.
 [19] P. Hall. The Edmonton notes on nilpotent groups. Queen Mary College Mathematics Notes. Mathematics Department, Queen Mary College, London, 1969.
 [20] D. F. Holt, B. Eick, and E. A. O’Brien. Handbook of computational group theory. Discrete Mathematics and its Applications (Boca Raton). Chapman & Hall/CRC, Boca Raton, FL, 2005.
 [21] J. Hughes and A. Tannenbaum. Lengthbased attacks for certain group based encryption rewriting systems, workshop seci02 sécurité de la communication sur internet. 2002.
 [22] G. Ivanyos, L. Sanselme, and M. Santha. An efficient quantum algorithm for the hidden subgroup problem in nil2 groups. In LATIN 2008: Theoretical Informatics, pages 759–771. Springer, 2008.
 [23] D. Kahrobaei and B. Khan. Nis056: A noncommutative generalization of ElGamal key exchange using polycyclic groups. In IEEE Globecom 2006, pages 1–5, Nov 2006.
 [24] D. Kahrobaei and C. Koupparis. Noncommutative digital signatures using noncommutative groups. Groups, Complexity, Cryptology, 4:377–â384, 2012.
 [25] K. H. Ko, S. J. Lee, J. H. Cheon, J. W. Han, J. Kang, and C. Park. New publickey cryptosystem using braid groups. Advances in cryptology, CRYPTO 2000 (Santa Barbara, CA), LNCS, vol. 1880, pages 166–183, 2000.
 [26] M. Kotov and A. Ushakov. Analysis of a certain polycyclicgroupbased cryptosystem. Journal of Mathematical Cryptology, 9(3):161–167, 2015.
 [27] C. R. LeedhamGreen and L. H. Soicher. Collection from the left and other strategies. J. Symbolic Comput., 9(56):665–675, 1990. Computational group theory, Part 1.
 [28] E. Lo and G. Ostheimer. A practical algorithm for finding matrix representations for polycyclic groups. J. Symbolic Comput., 28(3):339–360, 1999.
 [29] A. Mal’cev. On homomorphisms onto finite groups. Trans. Amer. Math. Soc, 119:67–79, 1983.
 [30] J. Milnor. Growth of finitely generated solvable groups. J. Differential Geom., 2(4):447–449, 1968.
 [31] A. D. Myasnikov and A. Ushakov. Lengthbased attack and braid groups: cryptanalysis of AnshelAnshelGoldfeld keyexchange protocol. PKC 2007, LNCS 4450, pages 76–88, 2007.
 [32] A. G. Myasnikov and V. Roman’kov. A linear decomposition attack. Groups Complexity Cryptology, 7(1):81–94, 2015.
 [33] A. G. Myasnikov, V. Shpilrain, A. Ushakov, and N. Mosina. Noncommutative cryptography and complexity of grouptheoretic problems, volume 177. American Mathematical Society Providence, RI, USA, 2011.
 [34] A. G. Myasnikov and A. Ushakov. Random subgroups and analysis of the lengthbased and quotient attacks. Journal of Mathematical Cryptology 2(1), pages 29–61, 2008.
 [35] W. Nickel. Matrix representations for torsionfree nilpotent groups by Deep Thought. J. Algebra, 300(1):376–383, 2006.
 [36] V. Remeslennikov. Conjugacy in polycyclic groups. Algebra and Logic, 8(6):404–411, 1969.
 [37] V. Roman’kov. The twisted conjugacy problem for endomorphisms of polycyclic groups. Journal of Group Theory, 13(3):355–364, 2010.
 [38] D. Segal. Decidable properties of polycyclic groups. Proc. London Math. Soc, 3:61–497, 1990.
 [39] P. Shor. Algorithms for quantum computation: Discrete logarithms and factoring. In Foundations of Computer Science, 1994 Proceedings., 35th Annual Symposium on, pages 124–134. IEEE, 1994.
 [40] V. Shpilrain. Search and witness problems in group theory. Groups–Complexity–Cryptology, 2(2):231–246, 2010.
 [41] V. Shpilrain and A. Ushakov. The conjugacy search problem in public key cryptography: unnecessary and insufficient. Applicable Algebra in Engineering, Communication and Computing, 17(34):285–289, 2006.
 [42] V. Shpilrain and A. Ushakov. An authentication scheme based on the twisted conjugacy problem. In Applied Cryptography and Network Security, pages 366–372. Springer, 2008.
 [43] V. Shpilrain and G. Zapata. Using the subgroup membership search problem in public key cryptography. Contemporary Mathematics, 418:169, 2006.
 [44] B. Tsaban. Polynomialtime solutions of computational problems in noncommutativealgebraic cryptography. Journal of Cryptology, 28:601–622, 2015.
 [45] B. Wehrfritz. Two remarks on polycyclic groups. Bulletin of the London Mathematical Society, 26(6):543–548, 1994.
 [46] J. Wolf. Growth of finitely generated solvable groups and curvature of Riemannian manifolds. Journal of Differential Geometry, pages 421–446, 1968.