Sorting by Transpositions is Difficult

# Sorting by Transpositions is Difficult

Laurent Bulteau, Guillaume Fertin, Irena Rusu
Laboratoire d’Informatique de Nantes-Atlantique (LINA), UMR CNRS 6241
Université de Nantes, 2 rue de la Houssinière, 44322 Nantes Cedex 3 - France
{Laurent.Bulteau, Guillaume.Fertin, Irena.Rusu}@univ-nantes.fr

Abstract. In comparative genomics, a transposition is an operation that exchanges two consecutive sequences of genes in a genome. The transposition distance, that is, the minimum number of transpositions needed to transform a genome into another, is, according to numerous studies, a relevant evolutionary distance. The problem of computing this distance when genomes are represented by permutations, called the Sorting by Transpositions problem, has been introduced by Bafna and Pevzner [BafnaPevzner95] in 1995. It has naturally been the focus of a number of studies, but the computational complexity of this problem has remained undetermined for 15 years.

In this paper, we answer this long-standing open question by proving that the Sorting by Transpositions problem is NP-hard. As a corollary of our result, we also prove that the following problem [Christie98] is NP-hard: given a permutation , is it possible to sort using permutations, where is the number of breakpoints of ?

## Introduction

Along with reversals, transpositions are one of the most elementary large-scale operations that can affect a genome. A transposition consists in swapping two consecutive sequences of genes or, equivalently, in moving a sequence of genes from one place to another in the genome. The transposition distance between two genomes is the minimum number of such operations that are needed to transform one genome into the other. Computing this distance is a challenge in comparative genomics, since it gives a maximum parsimony evolution scenario between the two genomes.

The Sorting by Transpositions problem is the problem of computing the transposition distance between genomes represented by permutations. Since its introduction by Bafna and Pevzner [BafnaPevzner95, BafnaPevzner98], the complexity class of this problem has never been established. Hence a number of studies [BafnaPevzner98, Christie98, GuPC99, HartmanS06, EliasH06, Benoit-GagneH07, FengZ07] aim at designing approximation algorithms or heuristics, the best known fixed-ratio algorithm being a 1.375-approximation [EliasH06]. Other works [GuyerHV97, Christie98, ErikssonEKSW01, Labarre06, EliasH06, Benoit-GagneH07] aim at computing bounds on the transposition distance of a permutation. Studies have also been devoted to variants of this problem, by considering, for example, prefix transpositions [DiasM02, Labarre08, ChitturiS08] (in which one of the blocks is a prefix of the sequence), or distance between strings [ChristieI01, CormodeM02, ShapiraS02, RadcliffeSW05, KolmanW06] (where multiple occurences of each element are allowed in the sequences), possibly with weighted or prefix transpositions [Qi2006, Bongartz06, AmirABLLPSV06, AmirAILP07, ChitturiS08].

In this paper, we address the long-standing issue of determining the complexity class of the Sorting by Transpositions problem, by giving a polynomial time reduction from SAT, thus proving the NP-hardness of this problem. Our reduction is based on the study of transpositions removing three breakpoints. A corollary of our result is the NP-hardness of the following problem, introduced by [Christie98]: given a permutation , is it possible to sort using permutations, where is the number of breakpoints of ?

## 1 Preliminaries

### 1.1 Transpositions and Breakpoints

In this paper, denotes a positive integer. Let , and be the identity permutation over . We consider only permutations of such that and are fixed-points. Given a word , a subword is a subsequence , where . A factor is a subsequence of contiguous elements, i.e. a subword with for every .

A transposition is an operation that exchanges two consecutive factors of a sequence. As we only work with permutations, it is defined as a permutation , which, once composed to a permutation , realise this operation (see Figure 1). The transposition is formally defined as follows.

###### Definition 1 (Transposition).

Given three integers such that , the transposition over is the following permutation (we write ):

 For any 0≤x

Note that the inverse function of is also a transposition. More precisely, .

The following two properties directly follow from the definition of a transposition:

###### Property 1.

Let be a transposition, , and be two integers such that . Then:

 τ(u)>τ(v) ⇔ i≤uτ−1(v) ⇔ i≤u
###### Property 2.

Let be the transposition , and write . For all , the values of and are the following:

 ∀x∉{i,q(j),k},τ(x−1)=τ(x)−1∀x∉{i,j,k},τ−1(x−1)=τ−1(x)−1
 τ(i−1)=τ(q(j))−1τ−1(i−1)=τ−1(j)−1τ(q(j)−1)=τ(k)−1τ−1(j−1)=τ−1(k)−1τ(k−1)=τ(i)−1τ−1(k−1)=τ−1(i)−1
###### Definition 2 (Breakpoints).

Let be a permutation of . If is an integer such that , then is an adjacency of , otherwise it is a breakpoint. We write the number of breakpoints of .

The following property yields that the number of breakpoints of a permutation can be reduced by at most 3 when a transposition is applied:

###### Property 3.

Let be a permutation and be a transposition (with ). Then, for all ,

Overall, we have .

###### Proof.

For all , we have:

 (x−1,x) adjacency of π ⇔π(x−1)=π(x)−1 ⇔π(τ(τ−1(x−1)))=π(τ(τ−1(x)))−1 ⇔π∘τ(τ−1(x)−1)=π∘τ(τ−1(x))−1 by Prop.\leavevmode\nobreak\ ??? ⇔(τ−1(x)−1,τ−1(x)) adjacency of π∘τ.

### 1.2 Transposition distance

The transposition distance of a permutation is the minimum number of transpositions needed to transform it into the identity. A formal definition is the following:

###### Definition 3 (Transposition distance).

Let be a permutation of . The transposition distance from to is the minimum value for which there exist transpositions , satisfying:

 π∘τk∘…∘τ2∘τ1=Idn

The decision problem of computing the transposition distance is the following:

Sorting by Transpositions Problem [BafnaPevzner95] Input: A permutation , an integer . Question: Is ?

The following property directly follows from Property 3, since for any the number of breakpoints of is .

###### Property 4.

Let be a permutation, then .

Figure 2 gives an example of the computation of the transposition distance.

## 2 3-Deletion and Transposition Operations

In this section, we introduce 3DT-instances, which are the cornerstone of our reduction from SAT to the Sorting by Transpositions problem, since they are used as an intermediate between instances of the two problems. We first define 3DT-instances and the possible operations that can be applied to them, then we focus on the equivalence between these instances and permutations.

### 2.1 3DT-instances

###### Definition 4 (3DT-instance).

A 3DT-instance of span is composed of the following elements:

• : an alphabet;

• : a set of (ordered) triples of elements of , partitioning (i.e. all elements are pairwise distinct, and );

• , an injection.

The domain of is the image of , that is the set .

The word representation of is the -letter word over (where ), such that for all , , and for , .

Two examples of 3DT-instances are given in Example 1. Note that such instances can be defined by their word representation and by their set of triples . The empty 3DT-instance, in which , can be written with a sequence of dots, or with the empty word .

###### TheoremExample 1 .

In this example, we define two 3DT-instances of span 6, and :

 I=a1 c2 b1 b2 c1 a2 with % T={(a1,b1,c1),(a2,b2,c2)}I′=\raisebox1.0pt$⋅$ b2 \raisebox1.0pt$⋅$ c2 \raisebox1.0pt$⋅$ a2 with T′={(a2,b2,c2)}

Here, has an alphabet of size 6, , hence is a bijection (, , , etc). The second instance, , has an alphabet of size 3, , with , , .

###### Property 5.

Let be a 3DT-instance of span with domain . Then

###### Proof.

We have since is an injection with image . The triples of partition so , and finally so . ∎

###### Definition 5.

Let be a 3DT-instance. The injection gives a total order over , written (or , if there is no ambiguity), defined by

 ∀σ1,σ2∈Σ,σ1≺Iσ2⇔ψ(σ1)<ψ(σ2) (1)

Two elements and of are called consecutive if there exists no element such that . In this case, we write (or simply ).

An equivalent definition is that if is a subword of the word representation of . Also, if the word representation of contains a factor of the kind (where represents any sequence of dots).

Using the triples in , we define a successor function over the domain :

###### Definition 6.

Let be a 3DT-instance with domain . The function is defined by:

 ∀(a,b,c)∈T,ψ(a) ↦ψ(b) ψ(b) ↦ψ(c) ψ(c) ↦ψ(a)

Function is a bijection, with no fixed-points, and such that is the identity over . In Example 1, we have:

### 2.2 3DT-steps

###### Definition 7.

Let be a 3DT-instance, and be a triple of . Write , , and . The triple is well-ordered if we have . In such a case, we write the transposition .

An equivalent definition is that is well-ordered iff one of , , is a subword of the word representation of . In Example 1, is well-ordered in : indeed, we have , and , so . The triple is also well-ordered in (), but not in : . In this example, we have and .

###### Definition 8 (3DT-step).

Let be a 3DT-instance with a well-ordered triple. The 3DT-step of parameter is the operation written , transforming into the 3DT-instance such that:

• with .

A 3DT-step has two effects on a 3DT-instance, as represented in Figure 3. The first is to remove a necessarily well-ordered triple from (hence from ). The second is, by applying a transposition to , to shift the position of some of the remaining elements. Note that a triple that is not well-ordered in can become well-ordered in , or vice-versa. In Example 1, can be obtained from via a 3DT-step: . Moreover, . A more complex example is given in Figure 4.

Note that a 3DT-step transforms the function into , restricted to , the domain of the new instance . Indeed, for all , we have

 succI′(ψ′(a)) =ψ′(b) =τ−1(ψ(b)) =τ−1(succI(ψ(a))) =τ−1(succI(τ(ψ′(a)))) =(τ−1∘succI∘τ)(ψ′(a))

The computation is similar for and .

###### Definition 9 (3DT-collapsibility).

A 3DT-instance is 3DT-collapsible if there exists a sequence of 3DT-instances such that

 Ik=I I0=ε

In Example 1, and are 3DT-collapsible, since . Another example is the 3DT-instance defined in Figure 4. Note that in the example of Figure 4, there are in fact two distinct paths leading to the empty instance.

### 2.3 Equivalence with the transposition distance

###### Definition 10.

Let be a 3DT-instance of span with domain , and be a permutation of . We say that and are equivalent, and we write , if:

With such an equivalence , the two following properties hold:

• The breakpoints of correspond to the elements of (see Property 6).

• The triples of breakpoints that may be resolved immediately by a single transposition correspond to the well-ordered triples of (see Figure 5 and Lemma 8).

###### Property 6.

Let be a 3DT-instance of span with domain , and be a permutation of , such that . Then the number of breakpoints of is .

###### Proof.

Let . By Definition 10, we have:

If , then , so is an adjacency of .

If , we write , so . Since has no fixed-point, we have , which implies . Hence, , and is a breakpoint of .

Consequently the number of breakpoints of is exactly , and by Property 5. ∎

With the following lemma, we show that the equivalence between a 3DT-instance and a permutation is preserved after a 3DT-step, see Figure 6.

###### Lemma 7.

Let be a 3DT-instance of span , and be a permutation of , such that . If there exists a 3DT-step , then and , where , are equivalent.

###### Proof.

We write the indices such that (i.e. , , ). Since is well-ordered, we have .

We have , with , , and . We write respectively and the domains of and . For all , we have

 v′∈L′ ⇔ ∃σ∈Σ−{a,b,c}, v′=τ−1(ψ(σ)) ⇔ τ(v′)∈L−{i,j,k}

We prove the 3 required properties (see Definition 10) sequentially:

• ,

• , let . Since , we have either , or . In the first case, we write (then ). By Property 2, is equal to , so . Hence,

 π′(v′−1)+1 =π(τ(τ−1(v)−1))+1 =π(u−1)+1 =π(v) by Def.\leavevmode\nobreak\ ???, since % v∈L and v=succI(u) =π′(v′)

In the second case, , we have

 π′(v′−1)+1 =π(τ(τ−1(v)−1))+1 =π(τ(τ−1(v−1)))+1 by Prop.\leavevmode\nobreak\ % ???, since v∉{i,j,k} =π(v−1)+1 =π(v) by Def.\leavevmode\nobreak\ ???, since % v∉L =π′(v′)

In both cases, we indeed have .

• Let be an element of . We write , , and . Then . Moreover, , hence .

 π′(u′−1)+1 =π(τ(τ−1(u)−1))+1 =π(τ(τ−1(u−1)))+1 by Prop.\leavevmode\nobreak\ % ???, since u∉{i,j,k} =π(u−1)+1 =π(v) by Def.\leavevmode\nobreak\ ???, since % v∈L and u=succ−1I(v) =π(τ(τ−1(v))) =π′(v′)

###### Lemma 8.

Let be a 3DT-instance of span , and a permutation of , such that . If there exists a transposition such that , then contains a well-ordered triple such that .

###### Proof.

We write , , and . Note that .

Let . For all , we have, by Property 3, that is an adjacency of iff is an adjacency of . Hence, since , we necessarily have that , and are breakpoints of , and , and are adjacencies of . We have

 π(i) =π(τ(i′)) =π′(i′) =π′(i′−1)+1 since (i′−1,i′) is an adjacency of π′ =π′(τ−1(i)−1)+1 =π′(τ−1(k−1))+1 by Prop.\leavevmode\nobreak% \ ??? =π(k−1)+1

Since and , by Definition 10, we necessarily have (where is the domain of ), and .

Using the same method with and , we obtain , and . Hence, contains one of the following three triples: , or . Writing this triple, we indeed have since . ∎

###### Theorem 9.

Let be a 3DT-instance of span with domain , and be a permutation of , such that . Then is 3DT-collapsible if and only if .

###### Proof.

We prove the theorem by induction on . For , necessarily and , and by Definition 10, (, and for all , ). In this case, is trivially 3DT-collapsible, and .

Suppose now , with , and the theorem is true for . By Property 6, we have , and by Property 4, .

Assume first that is 3DT-collapsible. Then there exist both a triple and a 3DT-instance such that , and that is 3DT-collapsible. Since , the size of is . By Lemma 7, we have , with . Using the induction hypothesis, we know that . So the transposition distance from to the identity is at most, hence exactly, .

Assume now that . We can decompose into , where is a transposition and a permutation such that . Since has breakpoints (Property 6), and has at most breakpoints (Property 4), necessarily removes breakpoints, and we can use Lemma 8: there exists a 3DT-step , where is a well-ordered triple and . We can now use Lemma 7, which yields . Using the induction hypothesis, we obtain that is 3DT-collapsible, hence is also 3DT-collapsible. This concludes the proof of the theorem. ∎

The previous theorem gives a way to reduce the problem of deciding if a 3DT-instance is collapsible to the Sorting by Transpositions problem. However, it must be used carefully, since there exist 3DT-instances to which no permutation is equivalent (for example, admits no permutation of such that ).

## 3 3DT-collapsibility is NP-Hard to Decide

In this section, we define, for any boolean formula , a corresponding 3DT-instance . We also prove that is 3DT-collapsible if and only if is satisfiable.

### 3.1 Block Structure

The construction of the 3DT-instance uses a decomposition into blocks, defined below. Some triples will be included in a block, in order to define its behavior, while others will be shared between two blocks, in order to pass information. The former are unconstrained, so that we can design blocks with the behavior we need (for example, blocks mimicking usual boolean functions), while the latter need to follow several rules, so that the blocks can conveniently be arranged together.

###### Definition 11 (l-block-decomposition).

An -block-decomposition of a 3DT-instance of span is an -tuple such that , for all , and . We write for , and .

Let . For , the factor of the word representation of is called the full block (it is a word over ). The subword of where every occurrence of is deleted is called the block .

For , we write if (equivalently, if appears in the word ). A triple is said to be internal if , external otherwise.

If is a transposition such that for all , , we write the -block-decomposition .

In the rest of this section, we mostly work with blocks instead of full blocks, since we are only interested in the relative order of the elements, rather than their actual position. Full blocks are only used in definitions, where we want to control the dots in the word representation of the 3DT-instances we define. Note that, for such that , the relation is equivalent to is a factor of .

###### Property 10.

Let be an -block-decomposition of a 3DT-instance of span , and be three integers such that (a) and (b) such that or (or both). Then for all , , and the -block-decomposition is defined.

###### Proof.

For any , we show that we cannot have . Indeed, implies (since ), and implies (since ). Hence implies , but contradicts both conditions and : hence the relation is impossible.

By Property 1, since for all , and does not hold, we have , which is sufficient to define . ∎

The above property yields that, if is a well-ordered triple of a 3DT-instance (), and is an -block-decomposition of , then is defined if is an internal triple, or an external triple such that one of the following equalities is satisfied: