Pattern-Avoiding Polytopes

Pattern-Avoiding Polytopes

Robert Davis and Bruce Sagan

Two well-known polytopes whose vertices are indexed by permutations in the symmetric group are the permutohedron and the Birkhoff polytope . We consider polytopes and , whose vertices correspond to the permutations in avoiding a set of patterns . For various choices of , we explore the Ehrhart polynomials and -vectors of these polytopes as well as other aspects of their combinatorial structure.

For , we consider all subsets and are able to provide results in most cases. To illustrate, is a Pitman-Stanley polytope, the number of interior lattice points in is a derangement number, and the normalized volume of is the number of trees on vertices.

The polytopes seems much more difficult to analyze, so we focus on four particular choices of . First we show that the is exactly the Chan-Robbins-Yuen polytope. Next we prove that for any containing we have . Finally, we study and , where the tilde indicates that we choose vertices corresponding to alternating permutations avoiding the pattern . In both cases we use order complexes of posets and techniques from toric algebra to construct regular, unimodular triangulations of the polytopes. The posets involved turn out to be isomorphic to the lattices of Young diagrams contained in a certain shape, and this permits us to give an exact expression for the normalized volumes of the corresponding polytopes via the hook formula. Finally, Stanley’s theory of -partitions allows us to show that their -vectors are symmetric and unimodal.

Various questions and conjectures are presented throughout.

1. Introduction

Let denote the symmetric group on and . Let and . We say that contains the pattern if there is some substring of whose elements have the same relative order as those in . Alternatively, we view as standardizing to by replacing the smallest element of with , the next smallest by , and so on. If there is no such substring then we say that avoids the pattern . If , then we say avoids if avoids every element of . We will use the notation

Note this is not the avoidance class of which is the union of these sets over all .

A polytope is the convex hull of finitely many points, written . Equivalently, a polytope may be described as a bounded intersection of finitely many half-spaces. The dimension of is the dimension of its affine span. We think of vectors in as columns and use to denote the usual inner product of . An affine hyperplane determined by the equation for some is called supporting if for every . If is a supporting hyperplane, then the set is called a face of and is a subpolytope of . Faces of dimension are vertices, faces of dimension are called edges, and faces of dimension are called facets. Additionally, we say a polytope is lattice if each vertex is an element of . Lattice polytopes have long found connections with permutations, in particular via the permutohedron and Birkhoff polytope.

The permutohedron is defined as

We will often make no distinction between a permutation and its corresponding point in . This polytope was first described in [25] and has connections to the geometry of flag varieties as well as representations of . We refer to [37] for general background regarding permutohedra.

The Birkhoff polytope is the polytope

The Birkhoff-von Neumann Theorem states that the vertices of are the permutation matrices.

In this article, we describe a natural blending of pattern avoidance with the permutohedron and the Birkhoff polytope. Specifically, for any set of patterns , we define to be the subpolytope of obtained by taking the convex hull of those vertices corresponding to permutations in . The polytope is defined similarly. We study the Ehrhart polynomials and -vectors of these polytopes as well as other aspects of their combinatorial structure.

The rest of this paper is organized as follows. In Section 2 we review some basic notions about pattern avoidance and polytopes which will be needed throughout. Section 3 focuses on the permutohedron case . We first show in Proposition 3.2 that the action of a certain subgroup of the dihedral group of the square produces unimodularly equivalent polytopes. We then consider all possible and are able to provide results for most of the orbits of this action. Specific propositions are listed in Table 1. As a sampling, is a Pitman-Stanley polytope, the number of interior lattice points in is a derangement number, and the normalized volume of is the number of trees on vertices.

The -avoiding Birkhoff polytope appears to be much harder to analyze in general. So we concentrate on four specific examples. In Section 4, we show that is a polytope studied by Chan, Robbins, and Yuen. Next we prove that for any containing the permutations and we have . In Section 5 we begin our study of and , the tilde indicating that we choose vertices corresponding to alternating permutations avoiding the pattern . In both cases we use order complexes of posets and techniques from toric algebra to construct regular, unimodular triangulations of the polytopes. The posets involved turn out to be isomorphic to the lattices of Young diagrams contained in a certain shape, and this permits us to give an exact expression for the normalized volumes of the corresponding polytopes via the hook formula. Finally, in Section 6, Stanley’s theory of -partitions is applied to show that the -vectors of these two polytopes are symmetric and unimodal.

Various conjectures and questions are scattered through the paper.

2. Preliminaries

There are a number of concepts to which we refer throughout the paper. In this section, we collect the most frequent of these notions.

2.1. Diagrams, Wilf Equivalence, and Grid Classes

Let . Sometimes for clarity we will insert commas and write . The diagram of a permutation is the set of points with Cartesian coordinates for . An example diagram is given in Figure 1. When no confusion will result, we make no distinction between a permutation and its diagram. Diagrams of permutations provide an easy way to see how certain permutations can be related geometrically. For example, the diagrams of and are related by reflection across the line . With both the -avoiding permutohedra and -avoiding Birkhoff polytopes, many results will be true not only for the choice of in their statement, but also for certain other subsets of permutations whose diagrams are related to those in .

Figure 1. The diagram of the permutation .

Two permutations and are called Wilf equivalent, written , if for all . For example, any two permutations in are Wilf equivalent. This is indeed an equivalence relation. Although proving may be quite difficult, in some instances the Wilf equivalence of two permutations follows quickly from observing that their diagrams are related by a transformation in the dihedral group of the square.

Let , where is rotation counterclockwise by an angle of degrees and is reflection across a line of slope . A couple of these rigid motions have easy descriptions in terms of the one-line notation for permutations. If then its reversal is , and its complement is .

Note that for any , one has if and only if , and hence . For this reason, the equivalences induced by the dihedral action on a square are often referred to as the trivial Wilf equivalences.

Call polytopes and unimodularly equivalent if one can be taken into the other by an affine transformation whose linear part is representable by an matrix with integer entries and determinant . We will see in Propositions 3.2 and 4.2 that certain trivial Wilf equivalences imply unimodular equivalence of the corresponding polytopes.

In subsequent sections, it will be helpful to describe classes of permutations in the following way: Let be a matrix with entries in . We say that a permutation is -griddable in if the diagram of can be partitioned into rectangular regions using horizontal and vertical lines in such a way that

If contains at most one element, it may be considered as either increasing or decreasing. For example, if

then is -griddable, as demonstrated in Figure 2. For a particular matrix , the grid class of is the set of permutations that are -griddable. We will occasionally use grid classes to more conveniently describe the structure of permutations used as the vertices polytopes.

Figure 2. An -gridding of .

2.2. Ehrhart Polynomials and Volume

For a lattice polytope , consider the counting function , where is the -th dilate of . This function is a polynomial in , although not obviously so; it is called the Ehrhart polynomial of . In particular, two well-known theorems due to Ehrhart [14] and Stanley [29] imply that the Ehrhart series of ,

may be written in the form

for some nonnegative integers with , , and .

We say the polynomial is the -polynomial of and the vector of coefficients, , is the -vector of . The -vector of a lattice polytope is a fascinating invariant, and obtaining a general understanding of -vectors of lattice polytopes and their geometric/combinatorial implications is currently of great interest.

A standard result of Ehrhart theory is that the leading coefficient of gives the volume of . We note, though that when a polytope is not full-dimensional, some extra care is needed when discussing volume. Usual Euclidean volume would dictate that the volume of a polytope that is not full-dimensional is zero. However, we are typically interested in the relative volume, that is, the volume of the polytope with respect to the lattice where is the affine subspace spanned by . When does have full dimension, the notions of volume and relative volume coincide. Throughout this paper, “volume” is understood to mean the relative volume.

The normalized volume of a lattice polytope is , where is the usual relative volume of . A lattice simplex with vertex set is unimodular with respect to the lattice if it has smallest possible relative volume with respect to . If is not specified, then it is assumed that . Equivalently, is unimodular with respect to if the set of emanating vectors forms a -basis of . In particular, if is unimodular, then it has a normalized volume of . We refer to Section 5.4 of [5] for a more thorough discussion of these details.

3. Permutohedra

The permutohedron has been generalized in multiple ways, including the permuto-associahedron of Kapranov [17], which was first realized as a polytope by Reiner and Ziegler [24], and the generalized permutohedra studied by Postnikov [22]. Here, we study yet another generalization of the permutohedron by looking at from the perspective of pattern avoidance.

Definition 3.1.

Let and define

to be the -avoiding permutohedron. If then we write for .

For example, if , then . If then, as previously remarked, where is the th Catalan number, so has a Catalan number of vertices.

Proposition 3.2.

If , then is unimodularly equivalent to for any . So their face lattices, volumes, and Ehrhart series are all equal.


For ease of notation, we prove this in the case that . The general demonstration is similar.

From the discussion above, is the image of under the map , where and the are the standard unit column vectors. Since is a permutation matrix, this is a unimodular transformation.

Also, is the image of under the map

which is again clearly unimodular. Finally, notice that and so gives rise to a unimodular equivalence as well. ∎

It turns out that two permutations and may be trivially Wilf equivalent without and being unimodularly equivalent. An explicit example is and : although these are related by a -degree rotation, one can compute that, while has facets, only has .

Proposition 3.2 allows us to choose more efficiently; a summary of the choices of leading to potentially distinct , and the corresponding results, are given in Table 1. Certain entries in the table have no corresponding result or conjecture provided; this is because no clear structure of is apparent in these cases. For example, one may verify that has facets whereas has only , so and are generally not unimodularly equivalent. In neither case, though, are there clear conjectures to be made.

Relevant result(s) for
Theorem 3.15
Conjecture 3.18
Proposition 3.8
Proposition 3.21
Proposition 3.19
Proposition 3.22
Conjecture 3.20
Proposition 3.23
Proposition 3.23
Proposition 3.23
Table 1. The choices of that result in unimodularly distinct , and references to the results proven about them.

3.1. Avoiding Two Patterns in

We begin by noting that if then for . This is because of the Erdős-Szekeres theorem which states that any permutation in contains either an increasing subsequence of length or a decreasing subsequence of length . The same is clearly true for any containing . So we do not need to consider polytopes for such avoidance classes.

The following result will be useful when considering in both the permutohedron and Birkhoff polytope cases. It follows easily from the proof of Proposition 5.2 in [13].

Lemma 3.3.

The permutations in are the intersection of the grid class for the matrix

with .

Proposition 3.4.

The polytope is a rectangular parallelepiped (parallelotope). Specifically, the polytope is contained in the hyperplane , and its facet-defining inequalities are


as ranges over .


Consider the polytope defined by the given inequalities and lying in the given hyperplane. Each inequality in (1) gives a pair of parallel faces of because of the absolute value signs. It is also easy to check that the normal vectors are pairwise orthogonal and also orthogonal to the vector which defines the hyperplane . Thus is an -dimensional parallelotope.

The polytope will have vertices. So to demonstrate that it suffices to prove that every is a vertex of . It follows from Lemma 3.3 that the elements of this avoidance class are characterized by the fact that for each , we have is either one greater than the largest previously-appearing entry or one less than the smallest previously-appearing entry. Note that if it is smaller, then satisfies , and if it is larger then satisfies . These equalities hold because the summands are exactly the integers in the first case and in the second. Since this is true for all , is a vertex of . ∎

Corollary 3.5.

The volume of is .


By the previous proposition, the volume of may be computed directly by choosing a base vertex, taking the product of the lengths of the edges incident to it, and then dividing by an appropriate factor to account for the relative volume. For the scaling factor, it is well-known that for a (measurable) subset and a linear function , with ,

where is the matrix for and volume is taken with respect to the usual Euclidean measure. In our case, a -basis for is for , so these vectors form the columns of . It is straightforward to check that where is the matrix with every entry . Furthermore, one easily sees that has one eigenvalue equal to (with corresponding eigenspace spanned by the all-ones vector) and the rest equal to (with corresponding eigenspace the subspace of vectors with coordinate sum zero). Thus . So to find the relative volume of , we must divide the usual -dimensional volume of by .

Now, a convenient choice of base vertex is the permutation . Using the hyperplane description of the previous result, this vertex is adjacent to the permutations for each . It is straightforward to compute that , so taking the product of these lengths and then dividing by yields as desired. ∎

Postnikov [22] defined generalized permutohedra and showed that they encompass associahedra, cyclohedra, Stanley-Pitman polytopes, and graphical zonotopes. So one could ask if is always a generalized permutohedron, since we would then immediately know its volume and, in some cases, its Ehrhart polynomial. However we will show that this is not the case for . To do this, we need a few more tools.

A fan in consists of a set of polyhedral cones in , each containing , such that

  • if and is a face of , then , and

  • for any and , is a face of both and .

Using the notation

we say a fan refines if and if each cone in is contained in a cone in . We note that the literature also uses the notation for .

Let and let be any polytope. Define

In other words, is the face of for which the linear form defined by is maximized. If is a face of a polytope , the normal cone of at is

In particular, if is a facet of , then is a ray. The collection of all , ranging over all faces of , is the normal fan of the polytope, and is denoted .

In our case, the inequalities of provide the rays of the normal fan for . We will compare this normal fan with a certain other fan, defined in the following way. The braid arrangement in is the set of hyperplanes . These hyperplanes partition the space into the Weyl chambers

where . The collection of these chambers and their lower-dimensional faces is the braid arrangement fan. The following result of Postnikov, Reiner, and Williams, allows us to see that does not fall into the class of generalized permutohedra.

Proposition 3.6 ([21, Proposition 3.2]).

A polytope in is a generalized permutohedron if and only if its normal fan, reduced by , is refined by the braid arrangement fan. ∎

Using the hyperplane description from Proposition 3.4, we can see immediately that the rays of are not all rays of the braid arrangement fan. Thus, the braid arrangement fan cannot be a refinement of . See Figure 3 for an example.

Figure 3. Viewed from , the rays are solid, while the rays of the braid arrangement fan are dashed.

The Ehrhart polynomial of is known to be , where is the number of forests with edges on vertex set (see Exercise 4.64(a) in [32]). The technique in this exercise can also be used to find the Ehrhart polynomial of . Our first step in this direction will use the following result, due to Stanley.

Theorem 3.7 ([31, Theorem 2.2]).

Suppose is a lattice zonotope, that is, can be written in the form

where each belongs to . The Ehrhart polynomial of is


where the sum ranges over all linearly independent subsets of and where is the greatest common divisor of all full minors of the matrix whose columns are the elements of . ∎

To state the next result elegantly we define, for nonnegative integers and , the falling factorial

Proposition 3.8.

The polytope has Ehrhart polynomial


From the half-space and hyperplane description given in Proposition 3.4, we can see that is, up to a translation by , the zonotope

where for . By applying the transformation , where is the upper-triangular matrix with in all positions along and above the diagonal, we see that is unimodularly equivalent to


for each . Note that the set of all is linearly independent.

We will now complete the proof using equation (2) on the basis. First, however, we need to set up some notation. For as in (2) we will use to stand for both the subset and the matrix whose columns are the elements of . For any family of subsets we define

We also let be the family of all -element subsets of and . So we will be done if we can prove that . In fact, we will show that the following recurrence relation holds:


It is easy to verify that satisfies the same recursion for . So induction on completes the proof once we have verified the base case . But is a single vertex so that which agrees with the fact that where the latter is the Kronecker delta.

Partition into the three subsets

From the definitions, one has

We now show that each of these summands equals the corresponding summand in (3).

The matrices in are the same as those for except with a last row of zeros. Clearly this row does not contribute any nonzero minors so , giving the first summand.

Now consider the minors of a matrix , letting be the submatrix of the minor. An example follows the proof to elucidate the method. If does not contain the last row of , then its last two columns are equal and . So the only contributing to are those whose last row is the final row of which is all zero except for a last entry of . It follows that where is obtained by removing the last row and column of . The possible which can appear are exactly those occurring in elements such that . Using the reasoning of the previous paragraph and complementation, we see that such contribute exactly to the desired sum. Thus .

Finally take so that ends with a sequence of at least two rows each of which has a sole nonzero entry at the end. Keeping the notation and reasoning of the previous paragraph, we see that if then must contain exactly one row from this final sequence. Let be the minors which can be obtained from all nonzero minors containing the last row of . Then for all we have where are exactly the nonzero minors of obtained by removing the last row and column of . So

Now repeat this process, but using the penultimate row of , giving minors with greatest common divisor . But and are relatively prime, so . Continuing in this way, we see that . Summing over all possible gives and completes the proof. ∎

To illustrate this demonstration, take and . Then a typical element of is

Considering the submatrix obtained by picking rows , , , and of and expanding around the last row we get where

Note that is also a submatrix of the matrix

and . The reader should now find it easy to construct a similar example for the argument concerning if need be.

A standard fact from Ehrhart theory states that the leading coefficient of is the volume of , so Corollary 3.5 is reaffirmed by the previous result. Moreover, knowing the Ehrhart polynomial allows us to deduce an interesting fact about the interior lattice points of .

Corollary 3.9.

The number of lattice points interior to is equal to the number of derangements in .


Let and be the interior of . By Proposition 3.8 and Ehrhart-Macdonald reciprocity [5, Theorem 4.1],

Evaluating at , we get

which is the well-known inclusion-exclusion formula for derangements. ∎

Question 3.10.

Is there a natural bijection between the interior points of and the derangements in ?

In the case of , the Ehrhart polynomial was simple enough to compute directly. Since the coefficients can be explicitly determined, one may also determine the -vector of by a change-of-basis, although there does not seem to be a simple formula for its components.

Although finding explicit formulas for -vectors is usually challenging in general, there are other methods for determining certain properties it might possess. A recent result due to Beck, Jochemko, and McCullough [3] states that lattice zonotopes always have a unimodal -vector. Thus the following result follows quickly from the proof of Proposition 3.8.

Corollary 3.11.

For all , is unimodal. ∎

Question 3.12.

For which -avoiding permutohedra is unimodal?

We will next consider a -avoiding permutohedron whose Ehrhart polynomial is easily computable due to results of Pitman and Stanley [20]. Given a sequence of nonnegative real numbers , there is a corresponding Pitman-Stanley polytope defined by

Pitman-Stanley polytopes are connected with multiple combinatorial objects. For example, recall that a polyhedral subdivision of a polytope is a collection of subpolytopes whose union is , and is a face of both and for all . Pitman and Stanley showed that has polyhedral subdivisions whose maximal elements of correspond to certain plane trees; can be expressed in terms of parking functions; the number of lattice points of can be expressed in terms of plane partitions of a particular shape. The key result for us is the following.

Theorem 3.13 (Pitman and Stanley, [20]).

Let be positive integers, and set . The Ehrhart polynomial of is

Before continuing, we need a little background. The face lattice of a polytope is the poset of its faces ordered by inclusion. Two polytopes are combinatorially equivalent if their face lattices are isomorphic. As proven in Theorem 19 of [20], whenever has positive entries, is combinatorially equivalent to an -cube.

Lemma 3.14.

When has positive entries, the vertices of are exactly the vectors constructed, component-wise from left to right, by either setting or setting , where is the previous nonzero entry of .


Since has positive entries, is a combinatorial cube, hence the set of facets may be partitioned into non-intersecting pairs. In particular, the pairs correspond to the hyperplanes and . Again, since is a combinatorial cube, a vertex will lie on exactly one of the facets of each pair. From these two facts, the conclusion follows. ∎

Theorem 3.15.

The polytope is a combinatorial cube with Ehrhart polynomial


We will show that is related to in such a way that its face lattice and Ehrhart polynomial are preserved. Then the theorem will follow from the statement just before Lemma 3.14, and by setting in Theorem 3.13.

We first need a description of the vertices of . By reversing the permutations in Proposition 4.2 of [13], we note that the diagram for a vertex of consists of a decreasing sequence of blocks where each block is the pattern for some . Define a function by

We claim that maps the vertices of to the vertices of . Indeed, suppose the first block of a vertex of is of the form . Then under this maps to the sequence with initial zeros. But, by Lemma 3.14, this is the prefix of a vertex of . Continuing in this way, we see that will indeed be a vertex of this Pitman-Stanley polytope. Reversing the argument shows that is, in fact, a bijection on the vertex sets.

Since is a subpolytope of the usual permutohedron, the projection to the first coordinates preserves the face lattice and Ehrhart polynomial, as does lattice translation. This verifies the claim in the first sentence of the proof. ∎

From the Ehrhart polynomial, we can immediately determine the volume and number of lattice points in the polytope.

Corollary 3.16.

The normalized volume of is and the number of lattice points it contains is the Catalan number


To calculate the normalized volume, one takes the leading coefficient of the Ehrhart polynomial in Theorem 3.15 and multiplies by since . To calculate the number of lattice points, one just plugs into this polynomial. ∎

We end this section with a question and a conjecture.

Question 3.17.

The normalized volume in Corollary 3.16 is just the number of trees on vertices and this quantity will also appear as a normalized volume in Proposition 3.22. And there are many combinatorial interpretations of the Catalan numbers. This raises the question of whether there is a combinatorial proof of Corollary 3.16 or Proposition 3.22.

The conjecture that follows makes a statement similar to that of Proposition 3.15. However, we have been unable to provide a proof.

Conjecture 3.18.

For all , is a combinatorial cube with normalized volume .

3.2. Avoiding Three or Four Patterns from

When contains least three or four patterns of , there are relatively few vertices of . Consequently, can be a farily simple object such as a simplex or line segment.

Proposition 3.19.

The Ehrhart polynomial for is and so is the Eulerian polynomial .


As noted in [8], it is implied by [12] that the simplex whose vertices are the set

has Ehrhart polynomial . Since the degree of the Ehrhart polynomial is the dimension of the polytope, is an -dimensional simplex. In particular, note that each satisfies the equation . So, projecting to by forgetting the last coordinate one obtains , which has the same Ehrhart polynomial as . Transforming by , where is the matrix with th column for and last column , results in the simplex whose vertices are and for .

As stated in the proof Proposition 16* from the paper of Simion and Schmidt [26], the permutations in are those obtained by inserting in all possible ways (between elements or at the beginning or end) into the decreasing sequence . So can also be obtained from by dropping the last coordinate and translating by . Since each of these operations is a unimodular transformation, has the same Ehrhart polynomial and -polynomial as , which are and , respectively. ∎

In the next conjecture, and are not related by reversal and complementation of patterns, so that Proposition 3.2 is not applicable. Yet the polytopes appear to be unimodularly equivalent. We have found no other instances of this phenomenon.

Conjecture 3.20.

For all , and are unimodularly equivalent.

Recall that the -dimensional standard simplex is the simplex whose vertices are the standard basis vectors of .

Proposition 3.21.

For all , is unimodularly equivalent to .


Again from the proof of [26, Proposition 16*] we see that the elements of are exactly the permutations of the form

for . Consider the transformation , defined by , where column of is for , column of is , and It is straightforward to check that , so that is a unimodular transformation, and that . ∎

Proposition 3.22.

For all , is a simplex with normalized volume .


Using the proof of Proposition 16* in [26] again, the elements of are

for . Consider the transformation , defined by