# Chromatic symmetric functions of hypertrees

## Abstract

The chromatic symmetric function of a hypergraph is the generating function for all colorings of so that no edge is monochromatic. When is an ordinary graph, it is known that is positive in the fundamental quasisymmetric functions , but this is not the case for general hypergraphs. We exhibit a class of hypergraphs — hypertrees with prime-sized edges — for which is -positive, and give an explicit combinatorial interpretation for the -coefficients of .

## 1 Introduction

In [11], Stanley defined the chromatic symmetric function of a graph , and since then this invariant has been an object of much study [3, 8, 9, 1, 4]. A proper coloring of an ordinary graph is a map so that if then . When is finite, we define the chromatic symmetric function to be a formal power series in the commuting indeterminates given by taking the sum over all proper colorings of where a vertex colored is assigned the weight . That is,

with the sum over all proper colorings of .

A formal power series (over, say, ) of bounded degree in is called a symmetric function if it remains the same after any permutation of its variables. The chromatic symmetric function is indeed symmetric, since the properness of a coloring is preserved under any permutation of the set of colors . It is natural, then, is to consider the expansion of in various bases of the ring of symmetric functions , and there are a number of conjectures and open problem concerning positivity of in these bases.

Our present interest is in the larger ring of quasisymmetric functions. A formal power series of bounded degree in the variables is called quasisymmetric if the coefficient of in is the same as the coefficient of in whenever and . The vector space of quasisymmetric functions of degree has dimension and has the basis of fundamental quasisymmetric functions indexed by subsets , defined by

with the sum over all weakly increasing sequences of positive integers with the restriction that if then . In what follows all the symmetric and quasisymmetric functions are homogeneous, so we will write without ambiguity. If is a symmetric function then it is also quasisymmetric, so we may consider the coefficients in the expansion . If each is nonnegative, we will say that is -positive. In [11], Stanley used the theory of -partitions to show that is always -positive; in that case the -coefficients count linear extensions of posets defined by acyclic orientations of .

In [13], Stanley presented a generalization of the chromatic symmetric function to hypergraphs. A hypergraph is a graph where the edges are allowed to contain more than two elements; that is, a hypergraph is a pair , finite, where is a family of subsets of called the hyperedges (or just edges) of with for each . We say is monochromatic under the coloring if maps all of to a single color , so that for all , and a coloring of is proper if no edge of is monochromatic under . It might seem more natural to require that each have a different color. However, as Stanley notes in [13], these colorings would be the same as the proper colorings of the ordinary graph obtained by replacing each hyperedge by the complete graph on , so nothing new would be gained by considering hypergraphs.

The chromatic symmetric function is then defined, as before, by

where the sum is taken over all proper colorings of .

Again is symmetric, but unlike in the case of ordinary graphs, is not always -positive. For example, if where and then

(1) |

is not -positive. The reader might observe that the coefficients in (1) sum to , and this is not a coincidence. The sum of the -coefficients in a chromatic symmetric function will always be where , and this can be seen by considering the coefficient of . Thus when is -positive, we might expect then to be able to write as a sum of fundamental quasisymmetric functions indexed by permutations, and this is what we proceed to do for a certain class of hypergraphs: the hypertrees with prime-sized edges. There are a number of closely-related definitions of hypertree occurring in the literature; we adopt the definition given in [6].

Let be a hypergraph. A path in is a nonempty sequence

where each with and the edges and vertices of the path are distinct, except that we allow . If and we say the path is a cycle. We say that a hypergraph is connected if there is a path from to for any given . A hypertree is a hypergraph that is connected and has no cycles. Thus in a hypertree there is a unique path between any two distinct vertices. A hypergraph is called linear if for any distinct edges . Hypertrees are linear, for if there are distinct for then there is a cycle . Figure 1 depicts a hypertree.

###### Theorem.

Let be a hypertree so that is a prime number for each edge . Then is -positive. In particular,

where and is the set of -descents of the permutation , to be defined in Section 4.

It is not true that is -positive whenever is linear. For example, is not -positive when consists of the edges . On the other hand, we guess (Conjecture A) that the primality condition is not necessary for -positivity, although our proof relies on primality in a crucial way. We also note that it is easy to extend Theorem 8 to disjoint unions of hypertrees, or hyperforests, but for simplicity we only consider the connected case.

In Section 2, we use a standardization procedure due to Gessel and Reutenauer [7] to show -positivity of when consists of a single prime-sized edge. In Section 3, we combine the result of Section 2 with the theory of -partitions due to Stanley [10] and Gessel[5] to show -positivity of for a hypertree with prime-size edges. In Section 4 we describe a combinatorial interpretation of the results in Section 3 by giving the definition of -descents and proving Theorem 8. We conclude in Section 5 by giving some conjectures and suggestions for further work.

## 2 The single edge case

If only has a single edge consisting of all of , so that , then a proper coloring of is any coloring that is not constant. Thus if then , where is the th power sum symmetric function

In this case it is not difficult to show that is -positive using standard results on symmetric functions. In fact, is Schur-positive, which implies -positivity. This in itself is not enough to show -positivity for other chromatic symmetric functions of hypergraphs. However, in this section we will prove -positivity of when is prime by exhibiting an explicit partition of the set of all nonconstant colorings where each set in the partition has a generating function that is a fundamental quasisymmetric function . We will see in Section 3 that if such a partition can be found for each edge in a hypertree then there is a similar partition of the set of proper colorings of that hypertree.

If is a finite set with , let be the set of permutations of realized as bijections ; if we write for . Given a permutation and a subset , let be the set of colorings satisfying the conditions

and when . Thus for any and , has the quasisymmetric generating function

Recall that the descent set of a permutation is the set of so that . The goal of this section is to prove the following fact.

###### Theorem 1.

Let be a set with prime and let be a cyclic permutation of . Then the set of nonconstant colorings is the disjoint union

Theorem 1 immediately gives the -expansion of the symmetric function when n is prime; we have

(2) |

If we think of each as a labeling of with the labels , identifying with , then is the same as when viewed as a permutation of the labels. Each cyclic permutation of the labels will appear times in this sum, so we have

(3) |

where the sum is taken over all cyclic permutations . The identity (3) was shown by Gessel and Reutenauer [7], and in fact if is prime then is both the generating function for primitive necklaces of length and the Frobenius chracteristic of the -representation given by the degree- multilinear part of the free Lie algebra on a set of size .

In the proof that follows it will be convenient to assume without loss of generality that and is the particular cyclic permutation for with . We think of a coloring as a word , where we write . To prove Theorem 1, we will use a method of obtaining a permutation from a nonconstant word when is prime due to Gessel and Reutenauer [7]. Let be a word that uses at least two distinct letters from , so that we do not have . Let be the rotation

of . The rotations need not be distinct in general. For example, if then . If is prime, however, this cannot occur and the rotations of are all distinct as long as is a nonconstant word.

Assuming is prime, define the cyclic standardization of , which we’ll denote , to be the permutation obtained by ordering these rotated words lexicographically: we say if is the unique permutation in so that whenever . That is, we find by setting when is the th smallest rotation of . For example, if then ; we have since is the least rotation of lexicographically, since is the next smallest, etc.

By the primality of , the set of words that are not constant is the disjoint union

Thus the proof of Theorem 1 is immediate from the following lemma.

###### Lemma 2.

Let be prime and let be a nonconstant word. Then if and only if where .

###### Proof.

First, suppose that . For any , suppose and ; then is the next largest rotation of in lexicographic order after , where we take , , etc. In particular, we must have , or as desired. Now suppose that is a descent of ; we must show that . We have ; that is, . Then we have

But we also know that , and the only way both of these lexicographic inequalities can occur is if .

Conversely, suppose that . To show that , we will show that implies . Since is prime, all rotations of are distinct, and so it is enough to show that implies ; we may also assume that and . We will proceed inductively. For any word , let be the truncation . We will show that for each , if we must have . If , the truncations are the single-character words ; and since .

Now suppose that the statement holds for . We will show that . By the argument for the base case, we know ; if we are done, so assume . Since , must be an ascent of ; that is, , or . By our inductive hypothesis, we must have . Then . ∎

## 3 Proof of F-positivity

Now we are in a position to show the -positivity of when is a hypertree with prime-sized edges. The primality gives us the decomposition described in Theorem 1 for each edge , and the hypertree structure will enable us to glue these decompositions together to get a similar decomposition of the set of all proper colorings of . The glue, in this case, is the theory of -partitions.

Given a poset on a vertex set , a mapping is a -partition if implies . If is the poset with the usual order, then a -partition is a sequence of increasing integers . This is equivalent to the usual definition of a partition of the integer . Traditionally a partition of an integer is written in descending order, so what we call -partitions were called reverse -partitions by Stanley [10, 14].

Suppose that is a bijection . In what follows it will be convenient to identify with the total order put on the vertices of where means that . A -partition is a -partition that has strict inequalities where the orders and disagree. That is, if and then , but if and then .

A linear extension of is an order-preserving bijection . The main result on -partitions we need is the following fact, sometimes called the Fundamental Theorem of -Partitions. See [12, Lemma 3.15.3] for a proof when -partitions are taken to be order-reversing; it is given without proof in [14, 7.19.4] for -partitions taken to be order-preserving as we do.

###### Theorem 3.

Let be a finite poset with and let be any bijection. Then the set of -partitions is exactly the disjoint union

where is as defined in Section 2, and the union is taken over all linear extensions of .

The key fact we will use about hypertrees is that posets on different edges are compatible with each other.

###### Lemma 4.

Let be a hypertree with . Suppose that each edge has an associated poset with vertex set and relation . Define the relation on by taking the transitive closure of all the relations , so that in if there is a chain . Then is a poset.

###### Proof.

Form a directed graph on by setting when there is an edge with and . Then is easily seen to be acyclic since is a hypertree, and any directed acyclic graph determines a poset after extending transitively. ∎

###### Theorem 5.

Let be a hypertree so that is prime for each . Then is -positive.

###### Proof.

Say . For each edge fix a particular bijection that is cyclic. Since each edge has prime, by Theorem 1 the set of nonconstant colorings is the disjoint union

(4) |

Let be the set of proper colorings of . A coloring of is proper if and only if each restriction is not constant, so we have

(5) |

where the union is taken over all -tuples with and

The union (5) is disjoint since each union (4) is: a coloring uniquely determines each since the restriction uniquely determines .

Given an edge and a bijection where , define a poset (actually a total order) on the vertex set by when , and let be the labeling . Then is the unique linear extension of , so by Theorem 3 the set of -partitions is exactly the set . Thus is the set of colorings so that is a -partition for each .

Fix a choice of for each edge . By Lemma 4 there are well-defined posets given by taking the transitive closure of the relations of the posets respectively. Let be any linear extension of . We claim that is exactly the set of -partitions. It is clear from the definitions that if is a -partition then is a -partition for each , so . Conversely, if it is not hard to see that is a -partition by repeatedly applying the “local conditions” that each is a -partition.

By rewriting (6) in a simpler form we can give an expression for as a sum of fundamental quasisymmetric functions indexed by permutations .

###### Corollary 6 (of the proof of Theorem 5).

Let be a hypertree. Given a permutation , define posets on so that when both belong to the same edge then iff and iff . Fix a linear extension of for each . Then

(7) |

###### Proof.

In Corollary 6 the choice of the linear extension is arbitrary. Every poset has a linear extension, and this fact is enough to prove -positivity. From a combinatorial standpoint, however, it would be desirable to find a specific choice of that is natural in some sense. In the next section we do so, giving a simple combinatorial interpretation to the -coefficients of .

## 4 Combinatorial interpretation

Before we can define the -descents alluded to in the introduction, we need to show the existence of a particularly nice ordering of the edges of a hypertree. The following lemma says that any hypertree may be constructed by adding one edge at a time, each new edge intersecting the others in a single vertex.

###### Lemma 7.

Let be a hypertree. Then there is an ordering of its edges so that with

(8) |

###### Proof.

It is enough to find an and so that for any with . Once such an is found, let with , . It is easy to check that is a hypertree with edges, so we may assume inductively that has an ordering of satisfying (8). Setting , we see that is the desired order of .

To find such an and , let be a path of maximal length in . We claim that , satisfies the desired property. Suppose that there is with ; then there is with . We must have for some , or else we would have a longer path where with . Say that is as large as possible. Then we have a cycle which violates the definition of a hypertree. ∎

In fact, the converse of Lemma 7 is easily seen to hold as well, so that the existence of edge-orderings satisfying characterizes hypertrees. From now on we will generally assume that is equipped with some choice of such an edge-ordering and our subsequent definitions are all based on this edge-ordering.

Suppose that is a hypertree with satisfying (8), and fix a choice of cyclic permutation for each edge . Suppose also that , so that is equipped with the order . Since is a hypertree, for each there is a unique path from to , where the vertices and edges in the path are all distinct. Let . Then we say that is an -descent if .

For example, let be the hypergraph in Fig. 2. The cyclic permutations are given by reading along the indicated direction in cycle notation, so that is with . The unique path from to is just since and are both contained in . Then , so is not an -descent. The unique path from to is given by and the edge with the smallest index occurring in this path is . Then and so is an -descent. Continuing, we find the -descents are .

Now let be a hypertree as before with an edge-ordering satisfying (8) and a cyclic permutation of each edge, but now allow to be an arbitrary finite set which we will think of as being unordered. Given a permutation , we will consider a labeling of , identifying with . We then denote the corresponding set of -descents by and call them the -descents of . With these definitions in hand, we state our main theorem.

###### Theorem 8.

Let be a hypertree so that is prime for each edge . Fix an ordering of the edges so with the property that for all , and also fix a choice of cyclic permutation of each edge . Then

where is the set of -descents of with respect to the chosen edge-ordering and cyclic permutations.

Note that in the case where consists of a single edge with a cyclic permutation , is exactly , so Theorem 8 reduces to Corollary 2.

To prove Theorem 8, we will need a systematic way of combining total orders together. Given totally ordered sets , where share a single element, say , we define to be the total order of the union given by “inserting” with its total order into the place of in . That is, is the unique total order agreeing with on so that when we have if and only if . Thus if consists of elements , with , and has elements , then is the total order of with

For example, if , are totally ordered sets then is the total order .

We now consider the total orders that arise from repeated insertion.

###### Lemma 9.

Let be a hypertree with so that (8) holds. Suppose there is a total order on each , and define a total order on by

Then for any distinct , if and only where

(9) |

is the unique path from to in and .

###### Proof.

We proceed by induction on the number of edges of . If has only one edge, the statement is trivial, so suppose that the statement holds for hypertrees with fewer than edges and that has exactly edges. Let be the hypertree with and , and let be the total order on given by

so that . If both and are in then we are done by the inductive hypothesis. Similarly if then there is nothing to show. So assume that and and let (9) be the path from to , so that . From the definition of the insertion we have if and only if . Then

is the unique path from to in and clearly since is the highest index of any edge in . Thus by our inductive hypothesis we see that is equivalent to . ∎