# Counting 3-Stack-Sortable Permutations

###### Abstract.

We prove a “Decomposition Lemma” that allows us to count preimages of certain sets of permutations under West’s stack-sorting map . As a first application, we give a new proof of Zeilberger’s formula for the number of -stack-sortable permutations in . Our proof generalizes, allowing us to find an algebraic equation satisfied by the generating function that counts -stack-sortable permutations according to length, number of descents, and number of peaks. This is also the first proof of this formula that generalizes to the setting of -stack-sortable permutations. Indeed, the same method allows us to obtain a recurrence relation for , the number of -stack-sortable permutations in . Hence, we obtain the first polynomial-time algorithm for computing these numbers. We compute for , vastly extending the terms of this sequence that were known before. We also prove the first nontrivial lower bound for , showing that it is at least . Our computations allow us to disprove a conjecture of Bóna, although we do not yet know for sure which one.

In fact, we can refine our methods to obtain a recurrence for , the number of -stack-sortable permutations in with descents and peaks. This allows us to gain a large amount of evidence supporting a real-rootedness conjecture of Bóna. Using part of the theory of valid hook configurations, we give a new proof of a -nonnegativity result of Brändén, which in turn implies an older result of Bóna. We then answer a question of the current author by producing a set such that has nonreal roots. We interpret this as partial evidence against the same real-rootedness conjecture of Bóna that we found evidence supporting. Examining the parities of the numbers , we obtain strong evidence against yet another conjecture of Bóna. We end with some conjectures of our own.

## 1. Introduction

### 1.1. The Stack-Sorting Map

We use the word “permutation” to refer to a permutation of a set of positive integers written in one-line notation. Let denote the set of permutations of the set . If is a permutation of length , then the normalization of is the permutation in obtained by replacing the -smallest entry in with for all . We say a permutation is normalized if it is equal to its normalization. A descent of a permutation is an index such that . A peak of is an index such that . Let and denote the number of descents of and the number of peaks of , respectively.

###### Definition 1.1.

Given , we say a permutation contains the pattern if there exist indices in such that the normalization of is . We say avoids if it does not contain . Let denote the set of normalized permutations that avoid the patterns . Let .

The study of permutation patterns is now a major area of research; it began with Knuth’s analysis of a certain “stack-sorting algorithm” [30]. In his dissertation, West [33] defined a deterministic variant of Knuth’s algorithm. This variant is a function, which we call the “stack-sorting map” and denote by , that sends permutations to permutations. The stack-sorting map has now been studied extensively [2, 7, 6, 5, 8, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 29, 32, 33, 35]. The reader seeking further historical background and motivation should see one of the references [2, 7, 18, 19, 20, 21, 22, 23].

To define the function , let us begin with an input permutation . At any point in time during this procedure, if the next entry in the input permutation is smaller than the entry at the top of the stack or if the stack is empty, the next entry in the input permutation is placed at the top of the stack. Otherwise, the entry at the top of the stack is annexed to the end of the growing output permutation. This process terminates when the output permutation has length , and is defined to be this output permutation. The following illustration shows that .

###### Definition 1.2.

We say a permutation is -stack-sortable if is an increasing permutation, where denotes the -fold iterate of . Let be the set of -stack-sortable permutations in , and let and . Let

Knuth simultaneously initiated the study of stack-sorting and the investigation of permutation patterns with the following theorem.

###### Theorem 1.1 ([30]).

A permutation is -stack-sortable if and only if it avoids the pattern . Furthermore,

where is the Catalan number.

In his dissertation, West conjectured a formula for , which Zeilberger later proved.

###### Theorem 1.2 ([35]).

We have

Combinatorial proofs of Zeilberger’s theorem emerged later in [16, 25, 26, 29]. Some authors have investigated the enumeration of -stack-sortable permutations according to various statistics [5, 10, 12, 25]. The articles [24] and [28] give different proofs that new combinatorial objects called “fighting fish” are counted by the numbers . The authors of [1] studied what they called “-point dominoes,” and they have found that there are such objects.

There is very little known about -stack-sortable permutations when . Úlfarsson [32] characterized -stack-sortable permutations in terms of new “decorated patterns,” but the characterization is too unwieldy to yield any additional information. The best known general upper bound for , which follows from a theorem of Stankova and West [7, Theorem 3.4], is the estimate

(1) |

The current author [20] showed that

(2) |

The limits in (2) are known to exist (see Section 6). Recently, Bóna has obtained a new proof of the first inequality in (2) using “stack words.” It also follows from Theorem 1.2 that

(3) |

When , we refer to (3) as a “trivial” lower bound for the growth rate of , even though it relies on the highly nontrivial enumeration of -stack-sortable permutations. Remarkably, (3) was the best known lower bound for for all until now.

Bóna [8] proved that the polynomial is symmetric and unimodal (see Section 7 for the relevant definitions). In fact, his proof actually shows that is symmetric and unimodal for every set . Brändén strengthened this result with the following theorem.

###### Theorem 1.3 ([14]).

If , then

In particular, is -nonnegative.

In the present article, we concern ourselves with the following four conjectures of Bóna. Recall that a sequence of positive numbers is called log-convex if is nondecreasing.

###### Conjecture 1.2 ([4]).

For every , the sequence is log-convex.

###### Conjecture 1.3 ([27]).

If is even, then is frequently odd. If is odd, then is rarely odd.

###### Conjecture 1.4 ([8]).

For all , the polynomial has only real roots.

###### Remark 1.1.

Bóna’s motivation for formulating Conjecture 1.1 came from the idea of encoding elements of as -uniform words over a -element alphabet (see [6] and [7] for more details). His motivation behind Conjecture 1.2 came from an observation that the sequences appear to be similar to the sequences that enumerate principle permutation classes, which he has also conjectured are log-convex. For example, Bóna has observed that his methods in [3] can be used to show that for fixed , the number of -stack-sortable permutations of length with components is monotonically decreasing as a function of . Similarly, his methods allow one to prove that the generating functions are not rational. Bóna formulated Conjecture 1.4 after observing that it holds when and when (it also holds when because this is equivalent to the case). Brändén [13] proved this conjecture in the cases and , but the remaining cases are still open.

Conjecture 1.3 requires some explanation. Using Bóna’s result that is symmetric, one can easily deduce that is even whenever is even. Therefore, it is natural to consider the parity of when is odd. Let be the number of integers with such that is odd. Let denote the Fibonacci number (with ). Using Theorems 1.1 and 1.2, one can show that and for all positive integers . Bóna [27] interpreted this as saying is rarely odd while is frequently odd, and this led him to formulate Conjecture 1.3. One could formalize this by saying that is rarely odd if and is frequently odd (although Bóna did not use this formalism). Bóna’s motivation behind Conjecture 1.3 also came from the idea of encoding -stack-sortable permutations with words.

### 1.2. Summary of Main Results

In Section 2, we formulate a “Decomposition Lemma,” which provides a new method for analyzing preimages of permutations under the stack-sorting map. We actually prove a stronger lemma, which we call the Refined Decomposition Lemma, that allows us to take the statistics and into account. In Section 3, we briefly review some formulas arising from the theory of new combinatorial objects called “valid hook configurations.” In Section 4, we use the Decomposition Lemma to give a new proof of Zeilberger’s formula for . We also use the Refined Decomposition Lemma to find an algebraic equation satisfied by the generating function of the numbers . This equation is new.

Our new proof of Zeilberger’s formula is the first one that generalizes to the setting of -stack-sortable permutations. In Section 5, we use the Refined Decomposition Lemma to prove a recurrence relation for the numbers . Specializing this theorem gives us a recurrence for , and specializing further gives a recurrence for . This yields the first polynomial-time algorithm for computing . According to Wilf [34], we have solved the problem of counting -stack-sortable permutations. More precisely, he would say that we have “-solved” this problem.

Before now, the values of were only known up to . Indeed, the only algorithm that was used to compute these numbers before now relied on a brute-force approach. Using our recurrence, we have generated the values of for . We have added these terms to sequence A134664 in the Online Encyclopedia of Integer Sequences [31]. There are two significant theoretical implications of these computations. First, we will see in Section 6 that Bóna’s Conjectures 1.1 and 1.2 cannot both be true. Thus, we have disproven a conjecture of Bóna, although we do not yet know with absolute certainty which one. Let us remark, however, that the data suggests very strongly that Conjecture 1.2 is true while Conjecture 1.1 is false. Furthermore, it appears that our recurrence coupled with sufficient computing time (and clever computing!) should allow one to completely disprove Conjecture 1.1. Second, we will prove that ; this is the first nontrivial lower bound for . Although there are multiple ways one could rigorously interpret Bóna’s Conjecture 1.3, we will see in Section 6 that every reasonable interpretation of the conjecture is likely to be false.

We have also computed the numbers for , allowing us to verify Conjecture 1.4 when and (see OEIS sequence A324916 [31]). In Section 7, we show that the formulas from Section 3 easily implies Brändén’s Theorem 1.3. We also provide a two-element set such that is not real-rooted. This provides a negative answer to the last part of Question 12.1 in [21], which we interpret as a small amount of evidence against Bóna’s Conjecture 1.4. Section 8 concludes the paper with a new conjecture about , several conjectures about the numbers (defined in Remark 1.1), and two new conjectures about unimodality and log-concavity.

Before we proceed, let us make one additional remark about the usefulness of the Decomposition Lemma that we prove in Section 2. In a forthcoming paper [17], we will use this lemma to prove a new lower bound for . We will also settle several conjectures of the current author from [21]. More precisely, we will complete the project of determining for every subset with the exception of the singleton set . This will allow us to enumerate a new permutation class, find a new example of an unbalanced Wilf equivalence, and prove a conjecture of Hossain concerning the so-called “Boolean-Catalan numbers.” Hence, one can even view the Decomposition Lemma as a bridge that allows one to use the stack-sorting map as a tool for proving results that were conjectured without any reference to stack-sorting.

## 2. The Decomposition Lemma

West [33] defined the fertility of a permutation to be , the number of preimages of under . He then went to great lengths to compute the fertilities of the permutations of the forms

Bousquet-Mélou [11] found a method for determining whether or not a given permutation is sorted, meaning that its fertility is positive. She then asked for a general method for computing the fertility of any given permutation. The current author achieved this in even greater generality in [19, 20, 21, 23] using new combinatorial objects called “valid hook configurations” ([23] is joint with Kravitz). In this section, we prove the Refined Decomposition Lemma and the Decomposition Lemma, which provide a new method for analyzing fertilities of permutations.

The plot of a permutation is the figure showing the points for all . For example, the image on the left in Figure 1 is the plot of . A hook of is obtained by starting at a point in the plot of , drawing a vertical line segment moving upward, and then drawing a horizontal line segment to the right that connects with a point . In order for this to make sense, we must have and . The point is called the southwest endpoint of the hook, while is called the northeast endpoint. Let be the set of hooks of with southwest endpoint . The right image in Figure 1 shows a hook of . This hook is in because its southwest endpoint is .

Define the tail length of a permutation , denoted , to be the smallest nonnegative integer such that . We make the convention that . The tail of is the sequence of points in the plot of . For example, the tail length of the permutation shown in Figure 1 is , and the tail of this permutation is . We say a descent of is tail-constrained if every hook in has its northeast endpoint in the tail of . The only tail-constrained descent of is .

Suppose is a hook of a permutation with southwest endpoint and northeast endpoint . Let and . The permutations and are called the -unsheltered subpermutation of and the -sheltered subpermutation of , respectively. For example, if and is the hook shown on the right in Figure 1, then and . In all of the cases we consider in this paper, the plot of lies completely below the hook in the plot of (it is “sheltered” by the hook ).

###### Lemma 2.1 (Refined Decomposition Lemma).

If is a tail-constrained descent of a permutation , then

###### Proof.

If the tail of is empty, then both sides of the desired equation are because and are empty. Hence, we may assume . Let . Given , we let be the entry that forces to leave the stack when we apply the stack-sorting procedure (described in the introduction) to . More precisely, is the leftmost entry that appears to the right of in and is larger than . Note that appears to the right of in . Because is tail-constrained, this means that the point is in the tail of . Given a point in the tail of , let be the set of permutations such that .

Now fix a point in the tail of , and let be the hook in with northeast endpoint . We will show that

from which the lemma will follow. We can write , where and . Suppose . Let us write . Because , it follows from the stack-sorting procedure that every entry in that is smaller than must appear to the left of in . This implies that every entry in that is smaller than is in . In particular, is in (we know that because is a descent of ). Now suppose is an entry in that is larger than . If is in , then we can appeal to the stack-sorting procedure again to see that must appear to the left of in . This is impossible, so every entry in is in . The stack-sorting procedure forces every entry in to be in , so every entry in that is not in must be an entry in . Furthermore, an entry in that is also in cannot appear to the left of one of the entries from in (otherwise, would appear to the right of one of the entries from in ). This proves that we can write , where is a permutation of the entries in . Moreover, every entry in is in .

Now let . One can verify that and . Let if is a descent of , and let otherwise. Because , the leftmost entry in is the leftmost entry in that appears to the right of in and is larger than (if no such entry exists, then is empty). Also, the rightmost entry in is less than . Combining these observations, we find that and .

We have shown how to take a permutation and decompose it into permutations and with and . We can easily reverse this procedure. Namely, if we are given and , we can write so that the leftmost entry in is the leftmost entry in that appears to the right of in and is larger than . We then recover by letting . ∎

###### Corollary 2.1 (Decomposition Lemma).

If is a tail-constrained descent of a permutation , then

###### Proof.

Set in Lemma 2.1. ∎

## 3. Fertility Formulas

The purpose of this brief section is to establish some terminology and state some formulas from [19] that we will use in Section 7. We will also use a very special consequence of Theorem 3.1 in Section 4 when we analyze the generating function of the numbers .

A composition of into parts is a -tuple of positive integers that sum to . For example, is a composition of into parts. Let denote the set of compositions of into parts. Let denote the Catalan number. Let

(4) |

Let

(5) |

The numbers are called Narayana numbers. They are given in the OEIS sequence A001263 and constitute the most common refinement of the Catalan numbers [31]. The polynomials are called Narayana polynomials. Among many other things, the Narayana numbers count binary plane trees with vertices and right edges. The numbers , which count binary plane trees with vertices and leaves, are given in the OEIS sequence A091894. Let be the number of binary plane trees with vertices, right edges, and leaves. Letting , we have

(6) |

This yields

from which one obtains

(7) |

Let

(8) |

so that

######
Theorem 3.1 ([19]^{1}^{1}1Strictly speaking, the first statement in Theorem 3.1 has not been stated explicitly before. However, the proofs of Corollary 5.1 and Theorem 5.2 in [19] immediately generalize to yield that statement.).

If and has exactly descents, then there exists a set such that

(9) |

In particular,

(10) |

and

(11) |

Thus,

(12) |

###### Remark 3.1.

If , then the above theorem, along with Theorem 1.1, tells us that

## 4. A New Proof of the Formula for

Recall from Section 2 the definition of the tail length of a permutation . Let (respectively, ) be the number of -stack-sortable permutations such that (respectively, ). Let

Because by Theorem 1.1, we can write

Suppose is such that (where ). Then is a tail-constrained descent of . The Decomposition Lemma (Corollary 2.1) tells us that is equal to the number of triples , where , , and . Choosing amounts to choosing the number such that the northeast endpoint of is . The permutation and the choice of determine the permutations and . On the other hand, the choices of and the permutations and uniquely determine . It follows that , which is the number of ways to choose an element of , is also the number of ways to choose , the permutations and , and the permutations and . Let us fix a choice of .

Because avoids , must be a permutation of the set , while must be a permutation of . Therefore, choosing and is equivalent to choosing their normalizations. The normalization of is in , while the normalization of is in (see Figure 2). Any element of can be chosen as the normalization of , and any element of can be chosen as the normalization of . Also, and have the same fertilities as their normalizations. Combining these facts, we find that the number of choices for and is . Similarly, the number of choices for and is . Hence,

(13) |

Let

Note that

by Theorem 1.1. Let be the generating function of the Catalan numbers. Because is the total number of -stack-sortable permutations in , our goal is to understand the generating function

At this point, we employ the techniques described in [9]; the reader seeking additional details can consult that article. There is a unique fractional power series (Puiseux series) such that and

(17) |

Indeed, we can compute the coefficients of one at a time from the equation (17) after we have initially computed the first few terms of via its combinatorial definition. We can now solve (17) for , use the standard Catalan functional equation , and clear denominators to obtain a polynomial

such that .

Let be the discriminant of with respect to the variable . A computer can explicitly compute this discriminant as , where

It follows from the method described in [9] that is a repeated root of . Since , we know that . Therefore, is a repeated root of . The discriminant of a polynomial with a repeated root must be . This means that , where is the discriminant of with respect to . Computing explicitly and ignoring extraneous factors, we find that , where

To complete our new proof of Theorem 1.2, we follow the proof of Proposition 5.2 in [11]. Namely, we consider the power series defined by . We then verify that and deduce that . Lagrange inversion then completes the proof that

The above argument generalizes as follows. Let