Noncancellable elements in type
affine Coxeter groups
Abstract.
Let be a Coxeter system and suppose that is fully commutative (in the sense of Stembridge) and has a reduced expression beginning (respectively, ending) with . If there exists such that and do not commute and (respectively, ) is no longer fully commutative, we say that is left (respectively, right) weak star reducible by with respect to . We say that a fully commutative element is noncancellable if it is irreducible under weak star reductions. In this paper, we classify the noncancellable elements in Coxeter groups of types and affine . In a sequel to this paper, the classification of the noncancellable elements play a pivotal role in inductive arguments used to prove the faithfulness of a diagrammatic representation of a generalized Temperley–Lieb algebra of type affine .
Key words and phrases:
Coxeter groups, noncancellable, star operations, heaps2000 Mathematics Subject Classification:
20F55, 06A07, 20C081. Introduction
Let be a Coxeter system with group and finite set of generating involutions . Our principal focus in this paper will be the infinite Coxeter group of type affine , denoted , and its finite subgroups of type . A wellknown result in the theory of Coxeter groups, known as Matsumoto’s Theorem, states that any two reduced expressions for are equivalent under the equivalence relation generated by braid relations. If is such that any two of its reduced expressions are equivalent by iterated commutations of commuting generators, is called fully commutative [20]. We denote the set of fully commutative elements of by . Fully commutative elements arise in several contexts and have many special properties relating to the study of the smoothness of Schubert varieties [6], Kazhdan–Lusztig polynomials [1, 11], and the decomposition of a Coxeter group into cells [13, 18].
Let . Suppose that has a reduced expression beginning with . Then we say that is left star reducible by with respect to to the shorter element provided that there exists such that and do not commute and has a reduced expression beginning with [10]. We make an analogous definition for right star reducible. The definition of star reducible is related to C.K. Fan’s definition of cancellable in [5, §4] and is also a special case of D. Kazhdan and G. Lusztig’s notion of a star operation, which is defined for arbitrary Coxeter systems in [17, §10.2].
We say that is star reducible if for every fully commutative , there exists a sequence such that each is left or right star reducible to and is equal to a product of commuting generators. It turns out that a Coxeter group of type is star reducible if and only if there is an even number of generators. However, Coxeter groups of type are star reducible regardless of the parity of the generating set [10, Theorem 6.3]. In a star reducible Coxeter group, products of commuting generators form the set of fully commutative elements that are irreducible under star reductions.
In this paper, we weaken the notion of star reducible and define the noncancellable elements, which include products of commuting generators. Let and suppose that has a reduced expression beginning with . We say that is left weak star reducible if (i) is star reducible by with respect to , and (ii) is no longer fully commutative. We make an analogous definition for right weak star reducible and define an element to be noncancellable (or weak star irreducible) if it is neither left or right weak star reducible.
The noncancellable elements of a Coxeter group are intimately related to the twosided cells of the generalized Temperley–Lieb algebra (in the sense of Graham [9]) associated to . The connection between the noncancellable elements and the twosided cells has been examined for types and in [5] and [7], respectively. This idea is also briefly touched upon for types , , and in [5]. Due to length considerations, we will not elaborate on the connection between the noncancellable elements and the twosided cells.
Our motivation for studying the noncancellable elements stems from the fact that computation involving the monomial basis elements of the generalized Temperley–Lieb algebra of that are indexed by noncancellable elements is “wellbehaved.” In fact, our classification of the noncancellable elements in a Coxeter group of type (Theorem 5.1.1) is a key component in the proof that establishes the faithfulness of a diagrammatic representation of the generalized Temperley–Lieb algebra of type , which is the focus of subsequent papers by the author.
In Section 2 of this paper, we establish our notation and introduce all of the necessary terminology. In Section 3, we explore some of the combinatorics of Coxeter groups of types and and introduce the type I and type II elements, which play a central role in this paper.
Section 4 is concerned with classifying the noncancellable elements in a Coxeter group of type (Theorem 4.2.1), which verifies Fan’s unproved claim in [5, §7.1] about the set of fully commutative elements in a Coxeter group of type having no generator appearing in the left or right descent set that can be left or right cancelled, respectively.
Using the classification of the type noncancellable elements, we prove the main result of this paper (Theorem 5.1.1), which classifies the infinitely many noncancellable elements in a Coxeter group of type . The proof of Theorem 5.1.1, as well as the preparatory lemmas, rely heavily on the notation of heaps that we develop in Section 2.4.
Lastly, in Section 6, we expand on our discussion of our motivation for classifying the noncancellable elements in a Coxeter group of type and briefly discuss future research.
This paper is an adaptation of chapters 1–5 of the author’s 2008 PhD thesis, titled A diagrammatic representation of an affine Temperley–Lieb algebra [4], which was directed by Richard M. Green at the University of Colorado at Boulder. However, some of the results presented here, especially those in Section 4, have new and streamlined arguments.
2. Preliminaries
2.1. Coxeter groups
A Coxeter system is pair consisting of a distinguished (finite) set of generating involutions and a group , called a Coxeter group, with presentation
where and . It turns out that the elements of are distinct as group elements, and that is the order of . Given a Coxeter system , the associated Coxeter graph is the graph with vertex set and edges labeled with for all . If , it is customary to leave the corresponding edge unlabeled. Given a Coxeter graph , we can uniquely reconstruct the corresponding Coxeter system . In this case, we say that the corresponding Coxeter system is of type , and denote the Coxeter group and distinguished generating set by and , respectively.
Given a Coxeter system , an expression is any product of generators from . The length of an element is the minimum number of generators appearing in any expression for the element . Such a minimum length expression is called a reduced expression. (Any two reduced expressions for have the same length.) A product with is called reduced if . Each element can have several different reduced expressions that represent it. Given , if we wish to emphasize a fixed, possibly reduced, expression for , we represent it in sans serif font, say , where each .
Matsumoto’s Theorem [8, Theorem 1.2.2] says that if , then every reduced expression for can be obtained from any other by applying a sequence of braid moves of the form
where , and each factor in the move has letters. The support of an element , denoted , is the set of all generators appearing in any reduced expression for , which is welldefined by Matsumoto’s Theorem. If , then we say that has full support.
Given a reduced expression for , we define a subexpression of to be any expression obtained by deleting some subsequence of generators in the expression for . We will refer to a consecutive subexpression of as a subword.
Let . We write
and
The set (respectively, ) is called the left (respectively, right) descent set of . It turns out that (respectively, ) if and only if has a reduced expression beginning (respectively, ending) with .
The main focus of this paper will be the Coxeter systems of types and , which are defined by the following Coxeter graphs, where .
We can obtain from by removing the generator and the corresponding relations [14, Chapter 5]. We also obtain a Coxeter group of type if we remove the generator and the corresponding relations. To distinguish these two cases, we let denote the subgroup of generated by and we let denote the subgroup of generated by . It is wellknown that is an infinite Coxeter group while and are both finite [14, Chapters 2 and 6].
2.2. Fully commutative elements
Let be a Coxeter system of type and let . Following Stembridge [20], we define a relation on the set of reduced expressions for . Let and be two reduced expressions for . We define if we can obtain from by applying a single commutation move of the form , where . Now, define the equivalence relation by taking the reflexive transitive closure of . Each equivalence class under is called a commutation class. If has a single commutation class, then we say that is fully commutative. By Matsumoto’s Theorem, an element is fully commutative if and only if no reduced expression for contains a subword of the form of length . The set of all fully commutative elements of is denoted by or .
Remark 2.2.1.
The elements of are precisely those whose reduced expressions avoid consecutive subwords of the following types:

for and ;

for or .
The fully commutative elements of and avoid the respective subwords above.
In [20], Stembridge classified the Coxeter groups that contain a finite number of fully commutative elements. According to [20, Theorem 5.1], contains an infinite number of fully commutative elements, while (and hence ) contains finitely many. There are examples of infinite Coxeter groups that contain a finite number of fully commutative elements. For example, Coxeter groups of type for are infinite, but contain only finitely many fully commutative elements [20, Theorem 5.1].
2.3. Noncancellable elements
The notion of a star operation was originally defined by Kazhdan and Lusztig in [16, §4.1] for simply laced Coxeter systems (i.e., for all ) and was later generalized to arbitrary Coxeter systems in [17, §10.2]. If is a pair of noncommuting generators for , then induces four partially defined maps from to itself, known as star operations. A star operation, when it is defined, respects the partition of the Coxeter group, and increases or decreases the length of the element to which it is applied by 1. For our purposes, it is enough to define star operations that decrease length by 1, and so we will not develop the full generality.
Suppose that is an arbitrary Coxeter system of type . Let and suppose that . We define to be left star reducible by with respect to to if there exists with . We analogously define right star reducible by with respect to . Observe that if , then is left (respectively, right) star reducible by with respect to if and only if (respectively, ), where the product is reduced. We say that is star reducible if it is either left or right star reducible by some .
We now introduce the concept of weak star reducible, which is related to Fan’s notion of cancellable in [5]. If , then is left weak star reducible by with respect to to if (i) is left star reducible by with respect to , and (ii) . Observe that (i) implies that and that . Furthermore, (ii) implies that . Also, note that we are restricting our definition of weak star reducible to the set of fully commutative elements. We analogously define right weak star reducible by with respect to . If is either left or right weak star reducible by some , we say that is weak star reducible. Otherwise, we say that is noncancellable or weak star irreducible (or simply irreducible).
Example 2.3.1.
Consider having reduced expressions and , respectively. We see that is left (respectively, right) weak star reducible by with respect to to (respectively, ), and so is not noncancellable. However, is noncancellable.
Remark 2.3.2.
We make a few observations regarding weak star operations.

If and (respectively, ), it is clear that (respectively, ) is still fully commutative. This implies that if is left or right weak star reducible to , then is also fully commutative.

It follows immediately from the definition that if is weak star reducible to , then is also star reducible to . However, there are examples of fully commutative elements that are star reducible, but not weak star reducible. For example, consider . We see that is star reducible, but not weak star reducible since and are still fully commutative for any . However, observe that in simply laced Coxeter systems (i.e., for all ), star reducible and weak star reducible are equivalent.

If , then is left weak star reducible by with respect to if and only if (reduced) when , or (reduced) when . Again, observe that the characterization above applies to and .
2.4. Heaps
Every reduced expression can be associated with a partially ordered set called a heap that will allow us to visualize a reduced expression while preserving the essential information about the relations among the generators. The theory of heaps was introduced in [23] by Viennot and visually captures the combinatorial structure of the Cartier–Foata monoid of [3]. In [20] and [21], Stembridge studied heaps in the context of fully commutative elements, which is our motivation here.
Although heaps will be useful for visualizing the arguments throughout the remainder of this paper, we will not exploit their full utility until Section 5, where we classify the noncancellable elements of type . In this section, we mimic the development found in [1], [2], and [20].
Let be a Coxeter system. Suppose is a fixed reduced expression for . As in [20], we define a partial ordering on the indices by the transitive closure of the relation defined via if and and do not commute. In particular, if and (since we took the transitive closure). This partial order is referred to as the heap of , where is labeled by . It follows from [20, Proposition 2.2] that heaps are welldefined up to commutativity class. That is, if and are two reduced expressions for that are in the same commutativity class, then the labeled heaps of and are equal. In particular, if is fully commutative, then it has a single commutativity class, and so there is a unique heap associated to .
Example 2.4.1.
Let be a reduced expression for . We see that is indexed by . As an example, since and the second and third generators do not commute. The labeled Hasse diagram for the unique heap poset of is shown below.
Let be a fixed reduced expression for . As in [1] and [2], we will represent a heap for as a set of lattice points embedded in . To do so, we assign coordinates (not unique) to each entry of the labeled Hasse diagram for the heap of in such a way that:

An entry with coordinates is labeled in the heap if and only if ;

An entry with coordinates is greater than an entry with coordinates in the heap if and only if .
Recall that a finite poset is determined by its covering relations. In the case of (and any straight line Coxeter graph), it follows from the definition that covers in the heap if and only if , , and there are no entries such that and . This implies that we can completely reconstruct the edges of the Hasse diagram and the corresponding heap poset from a lattice point representation. The lattice point representation of a heap allows us to visualize potentially cumbersome arguments. Note that our heaps are upsidedown versions of the heaps that appear in in [1] and [2] and several other papers. That is, in this paper entries on top of a heap correspond to generators occurring to the left, as opposed to the right, in the corresponding reduced expression. However, our convention aligns more naturally with the typical conventions of diagram algebras that are motivating the results of this paper.
Let be a reduced expression for . We let denote a lattice representation of the heap poset in described in the preceding paragraph. If is fully commutative, then the choice of reduced expression for is irrelevant, in which case, we will often write (note the absence of sans serif font) and we will refer to as the heap of .
Given a heap, there are many possible coordinate assignments, yet the coordinates for each entry will be fixed for all of them. In particular, two entries labeled by the same generator may only differ by the amount of vertical space between them while maintaining their relative vertical position to adjacent entries in the heap.
Let be a reduced expression for . If and are adjacent generators in the Coxeter graph with , then we must place the point labeled by at a level that is above the level of the point labeled by . Because generators that are not adjacent in the Coxeter graph do commute, points whose coordinates differ by more than one can slide past each other or land at the same level. To emphasize the covering relations of the lattice representation we will enclose each entry of the heap in a rectangle in such a way that if one entry covers another, the rectangles overlap halfway.
Example 2.4.2.
Let be as in Example 2.4.1. Then one possible representation for is as follows.
When is fully commutative, we wish to make a canonical choice for the representation by assembling the entries in a particular way. To do this, we give all entries corresponding to elements in the same vertical position and all other entries in the heap should have vertical position as high as possible. Note that our canonical representation of heaps of fully commutative elements corresponds precisely to the unique heap factorization of [23, Lemma 2.9] and to the Cartier–Foata normal form for monomials [3, 10]. In Example 2.4.2, the representation of that we provided is the canonical representation. When illustrating heaps, we will adhere to this canonical choice, and when we consider the heaps of arbitrary reduced expressions, we will only allude to the relative vertical positions of the entries, and never their absolute coordinates.
Given a canonical representation of a heap, it makes sense to refer to the th row of the heap, and we will do this when no confusion will arise. Note that for fully commutative elements, the first row of the heap corresponds to the left descent set. If , let denote the th row of the canonical representation for . We will write to mean that there is an entry occurring in the th row labeled by . If consists entirely of entries labeled by , then we will write .
Let have reduced expression and suppose and equal the same generator , so that the corresponding entries have coordinate in . We say that and are consecutive if there is no other occurrence of occurring between them in . In this case, and are consecutive in , as well.
Let be a reduced expression for . We define a heap to be a subheap of if , where is a subexpression of . We emphasize that the subexpression need not be a subword (i.e., a consecutive subexpression).
Recall that a subposet of is called convex if whenever in and . We will refer to a subheap as a convex subheap if the underlying subposet is convex.
Example 2.4.3.
As an example, let as in Example 2.4.1. Now, let be the subexpression of that results from deleting all but fifth, sixth, and last generators of . Then equals
and is a subheap of , but is not convex since there is an entry in labeled by occurring between the two consecutive occurrences of that does not occur in . However, if we do include the entry labeled by , then
is a convex subheap of .
From this point on, if there can be no confusion, we will not specify the exact subexpression that a subheap arises from.
The following fact is implicit in the literature (in particular, see the proof of [20, Proposition 3.3]) and follows easily from the definitions.
Proposition 2.4.4.
Let . Then is a convex subheap of if and only if is the heap for some subword of some reduced expression for . ∎
It will be extremely useful for us to be able to recognize when a heap corresponds to a fully commutative element in . The following lemma follows immediately from Remark 2.2.1 and is also a special case of [20, Proposition 3.3].
Lemma 2.4.5.
Let . Then cannot contain any of the following convex subheaps:
, , , , , ,
where and we use to emphasize that no element of the heap occupies the corresponding position. ∎
We conclude this section with an observation regarding heaps and weak star reductions. Let be a reduced expression for . Then is left weak star reducible by with respect to if and only if

there is an entry in labeled by that is not covered by any other entry; and

the heap contains one of the convex subheaps of Lemma 2.4.5.
Of course, we have an analogous statement for right weak star reducible.
3. The type I and type II elements of a Coxeter group of type
In this section, we explore some of the combinatorics of Coxeter groups of types and . Our immediate goal is to define two classes of fully commutative elements of that play a central role in the remainder of this paper. Most of these elements will turn out to be on our list of noncancellable elements appearing in Section 5.
3.1. The type I elements
Let . We define to be the maximum integer such that has a reduced expression of the form (reduced), where , , and is a product of commuting generators. Note that may be greater than the size of any row in the canonical representation of . Also, it is known that is equal to the size of a maximal antichain in the heap poset for [19, Lemma 2.9].
Definition 3.1.1.
Define the following elements of .

If , let
and
We also let .

If and , let

If and , let

If and , let

If and , let
If is equal to one of the elements in (1)–(5), then we say that is of type I.
The notation for the type I elements looks more cumbersome than the underlying concept. The notation is motivated by the zigzagging shape of the corresponding heaps. The index tells us where to start and the index tells us where to stop. The L (respectively, R) tells us to start zigzagging to the left (respectively, right). Also, (respectively, ) indicates the number of times we should encounter an end generator (i.e., or ) after the first occurrence of as we zigzag through the generators. If is an end generator, it is not included in this count. However, if is an end generator, it is included.
Example 3.1.2.
If , then
where we encounter an entry labeled by either or a combined total of times if and times if .
Every type I element is rigid, in the sense that each has a unique reduced expression. This implies that every type I element is fully commutative (there are no relations of any kind to apply). Furthermore, it is clear from looking at the heaps for the type I elements that if is of type I, then . Conversely, it follows by induction on that if for some , then must be of type I. Lastly, note that there are an infinite number of type I elements since there is no limit to the zigzagging that their corresponding heaps can do.
The discussion in the previous paragraph verifies the following proposition.
Proposition 3.1.3.
If is of type I, then is fully commutative with . Conversely, if , then is one of the elements on the list in Definition 3.1.1. ∎
3.2. The type II elements
It will be helpful for us to define . Then regardless of whether is odd or even, will always be the largest even number in . Similarly, will always be the largest odd number in .
Definition 3.2.1.
Define and . (Note that (respectively, ) consists of all of the odd (respectively, even) indices amongst .) Let and be of the same parity with . We define
Also, define
and
If is equal to a finite alternating product of and , then we say that is of type II. (It is important to point out that the corresponding expressions are indeed reduced.)
Example 3.2.2.
If is even, then
where the canonical representation has rows.
The next proposition follows immediately since the heaps of the type II elements avoid the impermissible configurations of Lemma 2.4.5.
Proposition 3.2.3.
Let be of type II. Then is fully commutative. Moreover, if is not equal to when is even, then .^{1}^{1}1The published version of this paper appearing in Int. Electron. J. Algebra 8, 2010 did not exclude the case when is equal to for even . The same error is contained in [4]. Thankfully, this error has no impact on the remaining results in this paper. ∎
It is quickly seen by inspecting the heaps for the type II elements that if is of type II, then is noncancellable. Note that if , then is the maximum value that can take. Furthermore, there are infinitely many type II elements. However, not every fully commutative element with value is of type II.
Note that if is even, then every with is not star reducible. This fact is implicit in [10] and is easily verified. It follows from our classification of the type noncancellable elements (see Theorem 5.1.1) that these elements are the only nonstar reducible elements in (with even) other than products of commuting generators; all other noncancellable elements are star reducible.
4. The type noncancellable elements
The goal of this section is to classify the noncancellable elements of . To accomplish this task, we shall make use of a normal form for reduced expressions in a Coxeter system of type .
4.1. Preparatory lemmas
Mimicking [12, §2.1], define
Then is a set of minimum length right coset representatives for the subgroup of , and for all and (see [14, §5.12]). It is an easy exercise to show that the elements of are given by
(One way this can be established is by working with the signed permutation representation of . Also, see [12, §2.1].)
Lemma 4.1.1.
Let . Then has a unique reduced decomposition , where each .
Proof.
See proof of Lemma 2.1.1 in [12]. ∎
We will refer to the unique reduced decomposition of Lemma 4.1.1 as the normal form factorization for .
The next two lemmas play a crucial role in the proof of Theorem 4.2.1.
Lemma 4.1.2.
Let have normal form factorization . If there exists such that , then for each , equals the identity or equals .
Proof.
For sake of a contradiction, assume otherwise. Choose the largest such that and is not equal to either the identity or . First, observe that we must have since . By how we chose , there must exist with such that . Choose the smallest such , so that for all . Then the only possibilities are that with , or with . Furthermore, is a subword of some reduced expression for . This implies that some reduced expression for would contain the subword , where the first occurrence of comes from while comes from . This violates being fully commutative. ∎
Lemma 4.1.3.
Let have normal form factorization and suppose that is noncancellable such that for all for some . Then is equal to or .
Proof.
Since does not appear in the support of and is not left weak star reducible, it must be the case that ; otherwise, is left weak star reducible by with respect to . Since is fully commutative and not right weak star reducible, it quickly follows that the only possibilities for in are or . ∎
4.2. Classification of the type noncancellable elements
The next theorem verifies Fan’s unproved claim in [5, §7.1] about the set of having no element of or that can be left or right cancelled, respectively.
Theorem 4.2.1.
Let . Then is noncancellable if and only if is equal to either a product of commuting generators, , or , where is a product of commuting generators with . We have an analogous statement for , where and are replaced with and , respectively.
Proof.
First, observe that if is noncancellable in for , then is also noncancellable in when considered as an element of the larger group. Also, we see that every element on our list is, in fact, noncancellable. It remains to show that our list is complete. We induct on the rank .
For the base case, consider . An exhaustive check verifies that the only noncancellable elements in are , , , and , which agrees with the statement of the theorem.
For the inductive step, assume that for all , our list is complete. Let and assume that is noncancellable with normal form factorization . If , then we are done by induction. So, assume that . In this case, , but for all . Since is not right weak star redudible, there are only two possibilities: (1) , or (2) .
Case (1): Suppose that . Then we may apply Lemma 4.1.2 and conclude that either (a) for all , or (b) for some . If we are in situation (a), then would be left weak star reducible by with respect to . Assume that (b) occurs and choose the largest such , so that while for . In this case, would be right weak star reducible by with respect to . Regardless, we contradict being noncancellable. Thus, we must be in case (2).
Case (2): Now, assume that , and for sake of a contradiction, assume that . This implies that
If , then would be right weak star reducible by with respect to . If for , then would be right weak star reducible by with respect to . Similarly, if with , then would be right weak star reducible by with respect to . Also, if , then is right weak star reducible by with respect to . The only remaining possibilities are: (a) , or (b) .
(a) Suppose that . By making repeated applications of Lemma 4.1.3, we can conclude that there exists such that . Choose the largest such . Then . By Lemma 4.1.2, is equal to either or . If , then would be left weak star reducible by with respect to . Yet, if , we would have right weak star reducible by with respect to . In either case, we contradict being noncancellable.
(b) Lastly, assume that . In this case, we can apply Lemma 4.1.2 and conclude that is equal to either or . If , then would be left weak star reducible by with respect to . On the other hand, if , then is right weak star reducible by with respect to . Again, we contradict being noncancellable.
Therefore, it must be the case that , which implies that . In this case, we can apply the induction hypothesis to and conclude that is one of the elements on our list (since would commute with all the elements in ). ∎
5. The type noncancellable elements
In this section, we will classify the noncancellable elements of .
5.1. Statement of theorem
The following theorem is the main result of this paper.
Theorem 5.1.1.
Let . Then is noncancellable if and only if is equal to one of the elements on the following list.

, where is a type noncancellable element and is a type noncancellable element with ;

, , , and ;

any type II element.
The elements listed in (i) include all possible products of commuting generators. This includes and , which are also included in (iii). The elements listed in (ii) are the type I elements having left and right descent sets equal to one of the end generators.
5.2. More preparatory lemmas
The proof of Theorem 5.1.1 requires several technical lemmas whose proofs rely heavily on the heap notation that we developed in Section 2.4.
Before proceeding, we make a comment on notation. When representing convex subheaps of for , we will use the symbol to emphasize the absence of an entry in this location of . It is important to note that the occurrence of the symbol implies that an entry from the canonical representation of cannot be shifted vertically from above or below to occupy the location of the symbol . If we enclose a region by a dotted line and label the region with , we are indicating that no entry of the heap may occupy this region.
We will make frequent use of the following lemma, which allows us to determine whether an element is of type I.
Lemma 5.2.1.
Let . Suppose that has a reduced expression having one of the following fully commutative elements as a subword:

,

,

,

.
Then is of type I.
Proof.
One quickly sees that if has any of the above reduced expressions as a subword, then must be of type I; otherwise, would contain one of the impermissible configurations of Lemma 2.4.5. ∎
The next two lemmas are generalizations of Lemma 5.3 in [10] and begin to describe the form that a noncancellable element that is not of type I can take. Recall from Section 2.4, that denotes the th row of the canonical representation of the heap of a fully commutative element.
Lemma 5.2.2.
Let with and suppose that is noncancellable and not of type I. If with , then this entry is covered by entries labeled by and .
Proof.
Note that our restrictions on and force . We proceed by induction.
For the base case, assume that . Then the entry in labeled by is covered by at least one of or . But since is not left weak star reducible, we must have both and occurring in .
For the inductive step, assume that the theorem is true for all for some . Suppose that with . Then at least one of or occur in . We consider two cases: (1) and (2) .
Case (1): Assume that in addition to , . Observe that this forces . Without loss of generality, assume that . (The case with is symmetric since the restrictions on imply that we may apply the induction hypothesis to either or .) By induction, the entry labeled by occurring in is covered by an entry labeled by and an entry labeled by . This implies that the entry labeled by occurring in must be covered by an entry labeled by ; otherwise, we produce one of the impermissible configurations of Lemma 2.4.5 corresponding to the subword . This yields our desired result.
Case (2): For the second case, assume that . Without loss of generality, assume that ; the case with can be handled by a symmetric argument. Then and this entry is covered by either (a) an entry labeled by , (b) an entry labeled by , or (c) both. If we are in situation (c), then we are done. For sake of a contradiction, assume that exactly one of (a) or (b) occurs.
First, assume that (a) occurs, but (b) does not. That is, and the entry labeled by that occurs in is not covered by an entry labeled by . Since and is fully commutative, it must be the case that while the entry labeled by occurring in is not covered by an entry labeled by .
For sake of a contradiction, assume that , so that is not the top row of the canonical representation for . Then we must have . Also, we cannot have ; otherwise, is left weak star reducible by with respect to . So, , which implies that the entry labeled by occurring in is covered. This entry cannot be covered by since is fully commutative. Therefore, we have . But by induction, this entry is covered by an entry labeled by and an entry labeled by . This produces one of the impermissible configurations of Lemma 2.4.5 corresponding to the subword , which contradicts being fully commutative. Thus, we must have .
Now, since is fully commutative and noncancellable, we must conclude that
forms the top rows of the canonical representation of , where for . Any other possibility either produces one of the impermissible configurations of Lemma 2.4.5 or violates not being right weak star reducible. Since is not of type I, this cannot be all of . The only possibility is that . Since is not right weak star reducible, this cannot be all of either. So, at least one of or occur in . We cannot have because is fully commutative. Thus, while is not. Again, since is not right weak star reducible, we must have while . Continuing with similar reasoning, we quickly see that
is a convex subheap of . But then by Lemma 5.2.1, is of type I, which is a contradiction. Therefore, we cannot have possibility (a) occurring while (b) does not.
The only remaining possibility is that (b) occurs, but (a) does not. That is, and , while the entry labeled by occurring in is not covered by an entry labeled by . Observe that the case is covered by an argument that is symmetric to the argument made above when we assumed that (a) occurs, but (b) does not, where we take instead of . So, assume that . Then by induction, entries labeled by and