Sorting by Transpositions is Difficult
Abstract. In comparative genomics, a transposition is an operation that exchanges two consecutive sequences of genes in a genome. The transposition distance, that is, the minimum number of transpositions needed to transform a genome into another, is, according to numerous studies, a relevant evolutionary distance. The problem of computing this distance when genomes are represented by permutations, called the Sorting by Transpositions problem, has been introduced by Bafna and Pevzner [BafnaPevzner95] in 1995. It has naturally been the focus of a number of studies, but the computational complexity of this problem has remained undetermined for 15 years.
In this paper, we answer this longstanding open question by proving that the Sorting by Transpositions problem is NPhard. As a corollary of our result, we also prove that the following problem [Christie98] is NPhard: given a permutation , is it possible to sort using permutations, where is the number of breakpoints of ?
Introduction
Along with reversals, transpositions are one of the most elementary largescale operations that can affect a genome. A transposition consists in swapping two consecutive sequences of genes or, equivalently, in moving a sequence of genes from one place to another in the genome. The transposition distance between two genomes is the minimum number of such operations that are needed to transform one genome into the other. Computing this distance is a challenge in comparative genomics, since it gives a maximum parsimony evolution scenario between the two genomes.
The Sorting by Transpositions problem is the problem of computing the transposition distance between genomes represented by permutations. Since its introduction by Bafna and Pevzner [BafnaPevzner95, BafnaPevzner98], the complexity class of this problem has never been established. Hence a number of studies [BafnaPevzner98, Christie98, GuPC99, HartmanS06, EliasH06, BenoitGagneH07, FengZ07] aim at designing approximation algorithms or heuristics, the best known fixedratio algorithm being a 1.375approximation [EliasH06]. Other works [GuyerHV97, Christie98, ErikssonEKSW01, Labarre06, EliasH06, BenoitGagneH07] aim at computing bounds on the transposition distance of a permutation. Studies have also been devoted to variants of this problem, by considering, for example, prefix transpositions [DiasM02, Labarre08, ChitturiS08] (in which one of the blocks is a prefix of the sequence), or distance between strings [ChristieI01, CormodeM02, ShapiraS02, RadcliffeSW05, KolmanW06] (where multiple occurences of each element are allowed in the sequences), possibly with weighted or prefix transpositions [Qi2006, Bongartz06, AmirABLLPSV06, AmirAILP07, ChitturiS08].
In this paper, we address the longstanding issue of determining the complexity class of the Sorting by Transpositions problem, by giving a polynomial time reduction from SAT, thus proving the NPhardness of this problem. Our reduction is based on the study of transpositions removing three breakpoints. A corollary of our result is the NPhardness of the following problem, introduced by [Christie98]: given a permutation , is it possible to sort using permutations, where is the number of breakpoints of ?
1 Preliminaries
1.1 Transpositions and Breakpoints
In this paper, denotes a positive integer. Let , and be the identity permutation over . We consider only permutations of such that and are fixedpoints. Given a word , a subword is a subsequence , where . A factor is a subsequence of contiguous elements, i.e. a subword with for every .
A transposition is an operation that exchanges two consecutive factors of a sequence. As we only work with permutations, it is defined as a permutation , which, once composed to a permutation , realise this operation (see Figure 1). The transposition is formally defined as follows.
Definition 1 (Transposition).
Given three integers such that , the transposition over is the following permutation (we write ):
Note that the inverse function of is also a transposition. More precisely, .
The following two properties directly follow from the definition of a transposition:
Property 1.
Let be a transposition, , and be two integers such that . Then:
Property 2.
Let be the transposition , and write . For all , the values of and are the following:
Definition 2 (Breakpoints).
Let be a permutation of . If is an integer such that , then is an adjacency of , otherwise it is a breakpoint. We write the number of breakpoints of .
The following property yields that the number of breakpoints of a permutation can be reduced by at most 3 when a transposition is applied:
Property 3.
Let be a permutation and be a transposition (with ). Then, for all ,
Overall, we have .
Proof.
For all , we have:
∎
1.2 Transposition distance
The transposition distance of a permutation is the minimum number of transpositions needed to transform it into the identity. A formal definition is the following:
Definition 3 (Transposition distance).
Let be a permutation of . The transposition distance from to is the minimum value for which there exist transpositions , satisfying:
The decision problem of computing the transposition distance is the following:
Sorting by Transpositions Problem [BafnaPevzner95] Input: A permutation , an integer . Question: Is ?
The following property directly follows from Property 3, since for any the number of breakpoints of is .
Property 4.
Let be a permutation, then .
Figure 2 gives an example of the computation of the transposition distance.
2 3Deletion and Transposition Operations
In this section, we introduce 3DTinstances, which are the cornerstone of our reduction from SAT to the Sorting by Transpositions problem, since they are used as an intermediate between instances of the two problems. We first define 3DTinstances and the possible operations that can be applied to them, then we focus on the equivalence between these instances and permutations.
2.1 3DTinstances
Definition 4 (3DTinstance).
A 3DTinstance of span is composed of the following elements:

: an alphabet;

: a set of (ordered) triples of elements of , partitioning (i.e. all elements are pairwise distinct, and );

, an injection.
The domain of is the image of , that is the set .
The word representation of is the letter word over (where ), such that for all , , and for , .
Two examples of 3DTinstances are given in Example 1. Note that such instances can be defined by their word representation and by their set of triples . The empty 3DTinstance, in which , can be written with a sequence of dots, or with the empty word .
TheoremExample 1 .
In this example, we define two 3DTinstances of span 6, and :
Here, has an alphabet of size 6, , hence is a bijection (, , , etc). The second instance, , has an alphabet of size 3, , with , , .
Property 5.
Let be a 3DTinstance of span with domain . Then
Proof.
We have since is an injection with image . The triples of partition so , and finally so . ∎
Definition 5.
Let be a 3DTinstance. The injection gives a total order over , written (or , if there is no ambiguity), defined by
(1) 
Two elements and of are called consecutive if there exists no element such that . In this case, we write (or simply ).
An equivalent definition is that if is a subword of the word representation of . Also, if the word representation of contains a factor of the kind (where represents any sequence of dots).
Using the triples in , we define a successor function over the domain :
Definition 6.
Let be a 3DTinstance with domain . The function is defined by:
Function is a bijection, with no fixedpoints, and such that is the identity over . In Example 1, we have:
2.2 3DTsteps
Definition 7.
Let be a 3DTinstance, and be a triple of . Write , , and . The triple is wellordered if we have . In such a case, we write the transposition .
An equivalent definition is that is wellordered iff one of , , is a subword of the word representation of . In Example 1, is wellordered in : indeed, we have , and , so . The triple is also wellordered in (), but not in : . In this example, we have and .
Definition 8 (3DTstep).
Let be a 3DTinstance with a wellordered triple. The 3DTstep of parameter is the operation written , transforming into the 3DTinstance such that:



with .
A 3DTstep has two effects on a 3DTinstance, as represented in Figure 3. The first is to remove a necessarily wellordered triple from (hence from ). The second is, by applying a transposition to , to shift the position of some of the remaining elements. Note that a triple that is not wellordered in can become wellordered in , or viceversa. In Example 1, can be obtained from via a 3DTstep: . Moreover, . A more complex example is given in Figure 4.
Note that a 3DTstep transforms the function into , restricted to , the domain of the new instance . Indeed, for all , we have
The computation is similar for and .
Definition 9 (3DTcollapsibility).
A 3DTinstance is 3DTcollapsible if there exists a sequence of 3DTinstances such that
2.3 Equivalence with the transposition distance
Definition 10.
Let be a 3DTinstance of span with domain , and be a permutation of . We say that and are equivalent, and we write , if:
With such an equivalence , the two following properties hold:

The breakpoints of correspond to the elements of (see Property 6).
Property 6.
Let be a 3DTinstance of span with domain , and be a permutation of , such that . Then the number of breakpoints of is .
Proof.
Let . By Definition 10, we have:
If , then , so is an adjacency of .
If , we write , so . Since has no fixedpoint, we have , which implies . Hence, , and is a breakpoint of .
Consequently the number of breakpoints of is exactly , and by Property 5. ∎
With the following lemma, we show that the equivalence between a 3DTinstance and a permutation is preserved after a 3DTstep, see Figure 6.
Lemma 7.
Let be a 3DTinstance of span , and be a permutation of , such that . If there exists a 3DTstep , then and , where , are equivalent.
Proof.
We write the indices such that (i.e. , , ). Since is wellordered, we have .
We have , with , , and . We write respectively and the domains of and . For all , we have
We prove the 3 required properties (see Definition 10) sequentially:

,

, let . Since , we have either , or . In the first case, we write (then ). By Property 2, is equal to , so . Hence,
In the second case, , we have
In both cases, we indeed have .

Let be an element of . We write , , and . Then . Moreover, , hence .
∎
Lemma 8.
Let be a 3DTinstance of span , and a permutation of , such that . If there exists a transposition such that , then contains a wellordered triple such that .
Proof.
We write , , and . Note that .
Let . For all , we have, by Property 3, that is an adjacency of iff is an adjacency of . Hence, since , we necessarily have that , and are breakpoints of , and , and are adjacencies of . We have
Since and , by Definition 10, we necessarily have (where is the domain of ), and .
Using the same method with and , we obtain , and . Hence, contains one of the following three triples: , or . Writing this triple, we indeed have since . ∎
Theorem 9.
Let be a 3DTinstance of span with domain , and be a permutation of , such that . Then is 3DTcollapsible if and only if .
Proof.
We prove the theorem by induction on . For , necessarily and , and by Definition 10, (, and for all , ). In this case, is trivially 3DTcollapsible, and .
Assume first that is 3DTcollapsible. Then there exist both a triple and a 3DTinstance such that , and that is 3DTcollapsible. Since , the size of is . By Lemma 7, we have , with . Using the induction hypothesis, we know that . So the transposition distance from to the identity is at most, hence exactly, .
Assume now that . We can decompose into , where is a transposition and a permutation such that . Since has breakpoints (Property 6), and has at most breakpoints (Property 4), necessarily removes breakpoints, and we can use Lemma 8: there exists a 3DTstep , where is a wellordered triple and . We can now use Lemma 7, which yields . Using the induction hypothesis, we obtain that is 3DTcollapsible, hence is also 3DTcollapsible. This concludes the proof of the theorem. ∎
The previous theorem gives a way to reduce the problem of deciding if a 3DTinstance is collapsible to the Sorting by Transpositions problem. However, it must be used carefully, since there exist 3DTinstances to which no permutation is equivalent (for example, admits no permutation of such that ).
3 3DTcollapsibility is NPHard to Decide
In this section, we define, for any boolean formula , a corresponding 3DTinstance . We also prove that is 3DTcollapsible if and only if is satisfiable.
3.1 Block Structure
The construction of the 3DTinstance uses a decomposition into blocks, defined below. Some triples will be included in a block, in order to define its behavior, while others will be shared between two blocks, in order to pass information. The former are unconstrained, so that we can design blocks with the behavior we need (for example, blocks mimicking usual boolean functions), while the latter need to follow several rules, so that the blocks can conveniently be arranged together.
Definition 11 (blockdecomposition).
An blockdecomposition of a 3DTinstance of span is an tuple such that , for all , and . We write for , and .
Let . For , the factor of the word representation of is called the full block (it is a word over ). The subword of where every occurrence of is deleted is called the block .
For , we write if (equivalently, if appears in the word ). A triple is said to be internal if , external otherwise.
If is a transposition such that for all , , we write the blockdecomposition .
In the rest of this section, we mostly work with blocks instead of full blocks, since we are only interested in the relative order of the elements, rather than their actual position. Full blocks are only used in definitions, where we want to control the dots in the word representation of the 3DTinstances we define. Note that, for such that , the relation is equivalent to is a factor of .
Property 10.
Let be an blockdecomposition of a 3DTinstance of span , and be three integers such that (a) and (b) such that or (or both). Then for all , , and the blockdecomposition is defined.
Proof.
For any , we show that we cannot have . Indeed, implies (since ), and implies (since ). Hence implies , but contradicts both conditions and : hence the relation is impossible.
By Property 1, since for all , and does not hold, we have , which is sufficient to define . ∎
The above property yields that, if is a wellordered triple of a 3DTinstance (), and is an blockdecomposition of , then is defined if is an internal triple, or an external triple such that one of the following equalities is satisfied: