We study an abstract notion of tree structure which lies at the common core of various tree-like discrete structures commonly used in combinatorics: trees in graphs, order trees, nested subsets of a set, tree-decompositions of graphs and matroids etc.
Unlike graph-theoretical or order trees, these tree sets can provide a suitable formalization of tree structure also for infinite graphs, matroids, and set partitions. Order trees reappear as oriented tree sets.
We show how each of the above structures defines a tree set, and which additional information, if any, is needed to reconstruct it from this tree set.
There are a number of concepts in combinatorics that express the tree-likeness of discrete111There are also non-discrete such concepts, such as -trees, which are not our topic here. structures. Among these are:
nested subsets, or bipartitions, of a set.
Other notions of tree-likeness, such as tree-decompositions of graphs or matroids, are modelled on these.
All these notions of tree-likeness work well in their own contexts, but sometimes less well outside them:
graph trees need vertices, which in some desired applications – even as close as matroids – may not exist;
order trees need additional poset structure which is more restrictive than the tree-likeness it implies;222Finite order trees, for example, correspond to rooted graph trees, but there is nothing in their definition from which we can abstract so that what remains corresponds to the underlying unrooted graph tree.
nested sets of bipartitions require a ground set that can be partitioned, which does not exist, say, in the case of tree-decompositions of a graph;
tree-decompositions of infinite graphs, which are modelled on graph trees, cannot describe separations that are limits of other separations, because graph trees do not have edges that are limits of other edges.
The purpose of this paper is to study an abstract notion of tree structure which is general enough to describe all these examples, yet substantial enough that each of these instances, in their relevant context, can be recovered from it.
We shall introduce this abstract notion of ‘tree sets’ formally in Section 2. It builds on a more general notion of ‘abstract separation systems’ developed in . That paper consists of no more than some basic notions and facts we need here anyway, and is thus required as preliminary reading.333Reference  started life as the preliminary sections of this paper, and it should be read first. The reason it was split off is that abstract separation systems have since been used in several other papers too, and will be in more to come. So it seemed sensible to have the basics collected together in one place. In a nutshell, a separation system will just be a poset with an order-reversing involution, and tree sets will be nested separation systems: sets in which every element is comparable with every other element or its inverse.
Tree-likeness has been modelled in many ways , and even the idea to formalize it in this abstract way is not new. Abstract nested separation systems as above were introduced by Dunwoody [9, 10] under the name of ‘protrees’, as an abstract structure for groups to act on, and used by Hundertmark  as a basis for structure trees of graphs and matroids that can separate their tangles and related substructures. They have not, however, been studied systematically – which is our purpose in this paper.
Although studying abstract tree sets may seem amply justified by their ubiquity in different contexts, there are two concrete applications that I would like to point out. The first of these is to order trees. These are often used in infinite combinatorics to capture tree structure wherever it arises. The reason they can do this better than graph-theoretical trees is that they may contain limit points, as those tree-like structures to be captured frequently do. However, order trees come with more information than is needed just to capture tree structure, which can make their use cumbersome. For a finite tree structure, for example, they correspond to a graph-theoretical tree together with the choice of a root. A change of root will change the induced tree order but not the underlying graph tree, which already captures that finite tree structure.
It turns out that abstract tree sets can provide an analogue of this also for infinite order trees: these will correspond precisely to the consistently oriented tree sets. Just as different choices of a root turn the same graph-theoretical tree into related but different order trees, different consistent orientations of an abstract trees set yield related but different order trees. Order trees thus appear as something like a category of ‘pointed tree sets’,444More precisely: tree sets plus a consistent orientation. The latter can be specified by a ‘point’ – i.e., an element of the set – only when the order tree has a least element. not only when their tree structure is represented by a graph but always, also when they have limit points. It thus becomes possible to ‘forget’ the ordering of an order tree but retain more than a set: the set plus exactly the information that makes it tree-like.
The application of tree sets that originally motivated this paper was one to graphs, as follows. In graph minor theory there are duality theorems saying that a given finite graph either has a certain highly connected substructure, such as a bramble, or if not then this is witnessed by a tree-decomposition showing that such a highly connected substructure cannot exist, because ‘there is no room for it’ [7, 13]. The edges of the decomposition tree then correspond to separations of this graph that form a tree set in our sense.
Conversely, a tree set of separations of a finite graph or matroid is always induced by a tree-decomposition in this way. Thus, in finite graphs and matroids, tree-decompositions and tree sets of separations amount to the same thing.
But for infinite graphs, tree sets of separations are more powerful: they need not come from tree-decompositions, since separations can have limits but edges in decomposition trees do not. This is why width duality theorems for infinite graphs, such as those in , require tree sets of separations, rather than tree-decompositions, to express the tree structures that witness the absence of highly connected substructures such as tangles or brambles. See  for more on this.
Highly cohesive substructures can be identified not only in graphs, but also in much more general combinatorial structures: all those that come with a sensible notion of ‘separation’, and hence give rise to abstract separation systems. Their highly cohesive substructures then take the form of tangles of these separation systems: orientations of their separations that are consistent in various ways (all including the basic consistency considered for tree sets in this paper) that define the differences between these types of tangle [4, 6, 7, 8]. We thus obtain a wealth of duality theorems for potentially very different combinatorial structures all based on the abstract tree sets studied in this paper.
In Section 2 we provide just the formal definitions needed to state our theorems; for all the basic facts about abstract separation systems that we shall need in our proofs we refer the reader to . In Sections 3, 4, 5 and 6, respectively, we show how abstract tree sets can be used to describe the tree structures of our earlier examples: of graph-theoretical trees, of order trees, of nested sets of bipartitions of a set, and of tree-decompositions of graphs and matroids. We shall also see how these representations of tree sets can be recovered from the tree sets they represent. Where relevant we shall point out how, conversely, abstract tree sets describe tree-like structures in these contexts that do not come from such examples: where tree sets provide not just a convenient common language for different kinds of tree structures but define new ones, including new ones that are needed for applications in traditional settings such as graphs and matroids .
2 Abstract separation systems and tree sets555This section is provided only to make this paper formally self-contained; I encourage the reader to read  instead of this section, or at least to refer to  in parallel.
A separation of a set is a set such that .777We can make further requirements here that depend on some structure on which is meant to separate. If is the vertex set of a graph , for example, we usually require that has no edge between and . But such restrictions will depend on the context and are not needed here; in fact, even the separations of a set defined here is just an example of the more abstract ‘separations’ we are about to introduce. The ordered pairs and are its orientations. The oriented separations of are the orientations of its separations. Mapping every oriented separation to its inverse is an involution that reverses the partial ordering
since the above is equivalent to . Informally, we think of as pointing towards and away from .
More generally, a separation system is a partially ordered set with an order-reversing involution *. Its elements are called oriented separations. An isomorphism between two separation systems is a bijection between their underlying sets that respects both their partial orderings and their involutions.
When a given element of is denoted as , its inverse will be denoted as , and vice versa. The assumption that * be order-reversing means that, for all ,
A separation is a set of the form , and then denoted by . We call and the orientations of . The set of all such sets will be denoted by . If , we call both and degenerate.
When a separation is introduced ahead of its elements and denoted by a single letter , we shall use and (arbitrarily) to refer to its elements. Given a set of separations, we write for the set of all the orientations of its elements. With the ordering and involution induced from , this is again a separation system.888When we refer to oriented separations using explicit notation that indicates orientation, such as or , we sometimes leave out the word ‘oriented’ to improve the flow of words. Thus, when we speak of a ‘separation ’, this will in fact be an oriented separation.
Separations of sets, and their orientations, are clearly an instance of this abstract setup if we identify with .
A separation is trivial in , and is co-trivial, if there exists such that as well as . We call such an a witness of and its triviality. If neither orientation of is trivial, we call nontrivial.
Note that if is trivial in then so is every . If is trivial, witnessed by , then by (1). Hence if is trivial, then cannot be trivial. In particular, degenerate separations are nontrivial.
There can also be separations with that are not trivial. But anything smaller than these is again trivial: if , then witnesses the triviality of . Separations such that , trivial or not, will be called small; note that, by (1), if is small then so is every .
The trivial oriented separations of a set , for example, are those of the form with and for some in the set considered. The small separations of are all those with .
A separation system is regular if it has no small elements. It is essential if it has neither trivial elements nor degenerate elements.
Note that all regular separation systems are essential.
The essential core of a separation system is the essential separation system obtained from by deleting all its separations that are degenerate, trivial, or co-trivial in .
An essential but irregular separation system can be made regular by deleting all pairs of the form from the relation viewed as a subset of : the triple , where if and only if and , is a regular separation system . We call it the regularization of .
A set of oriented separations is antisymmetric if for all : if does not contain the inverse of any of its nondegenerate elements. We call consistent if there are no distinct with orientations such that .
Two separations are nested if they have comparable orientations; otherwise they cross. Two oriented separations are nested if and are nested.999Terms introduced for unoriented separations may be used informally for oriented separations too if the meaning is obvious, and vice versa. We say that points towards , and points away from , if or .
In this informal terminology, two oriented separations are nested if and only if they are either comparable or point towards each other or point away from each other. And a set is consistent if and only if it does not contain orientations of distinct separations that point away from each other.
A set of separations is nested if every two of its elements are nested.
A tree set is a nested essential separation system. An isomorphism of tree sets is an isomorphism of separation systems that happen to be tree sets.
The essential core of a nested separation system is the tree set induced by .
A star (of separations) is a set of nondegenerate oriented separations whose elements point towards each other: for all distinct .
We allow . Note that stars of separations are nested. They are also consistent: if distinct lie in the same star we cannot have , since also by the star property.
A star is proper if, for all distinct , the relation required by the definition of ‘star’ is the only one among the four possible relations between orientations of distinct and : if but and and .
Our partial ordering on also relates its subsets, and in particular its stars: for we write if for every there exists some with . This relation is obviously reflexive and transitive, but in general it is not antisymmetric: if contains separations , then for we have (where denotes ‘ but not =’). However, it is antisymmetric on antichains, and thus in particular on proper stars .
We call a star proper in if it is proper and is not a singleton with co-trivial in . We shall call such stars co-trivial singletons.
When we speak of maximal proper stars in a separation system , we shall always mean stars that are -maximal in the set of stars that are proper in . Maximal stars in tree sets will play a key role in describing their structure.
We refer to  for examples and various properties of stars in separation systems that we shall use throughout this paper.
An orientation of a separation system , or of a set of separations, is a set that contains for every exactly one of its orientations . A partial orientation of is an orientation of a subset of : an antisymmetric subset of . We shall be interested particularly in consistent orientations.
Every consistent orientation of a regular separation system is the down-closure
in of the set of its maximal elements – provided that every element of lies below some maximal element (which can fail when is infinite). If is a tree set, then these sets are its maximal proper stars; we call them the splitting stars of .
See  for proofs of these assertions and further background needed later.
3 Tree sets from graph-theoretical trees
of all orientations of the edges of a tree form a regular tree set with respect to the involution and the natural partial ordering on : the ordering in which if and the unique – path in joins to . We call this is the edge tree set of . For every node of , we call the set
of edges at and oriented towards the oriented star at in .
The sets are the splitting stars of the edge tree set of .
Let be any tree. The down-closure in of a set is clearly a consistent orientation of whose set of maximal elements is precisely .
Conversely, let split . Then is the set of maximal elements of some consistent orientation of , and . In particular, unless (in which case the assertion is true), so has a maximal element . For every neighbour of , the maximality of in implies that and hence .
Thus, . As is closed down in and the down-closure of in orients all of , this down-closure equals and has as its set of maximal elements, giving as desired. ∎
Lemma 3.1 allows us to recover a tree from its edge tree set . Indeed, given just , let be the set of its splitting stars . Define a graph on by taking as its set of oriented edges and assigning to every edge the splitting star of containing it as its terminal node. These are well defined – i.e., every lies in a unique splitting star – by Lemma 3.1 and our assumption that . Then, clearly, the map is a graph isomorphism between and .
Our assumption above that is the edge tree set of some tree was used heavily in the argument above. And indeed, it cannot be omitted altogether: an arbitrary tree set need not be realizable as the edge tree set of a tree. Finite regular tree sets are, though, and our next aim is to prove this in Theorem 3.3.
Given a tree set , write for the set of its consistent orientations. Define a directed graph with edge set as follows. For each there is a unique in which is maximal, by [2, Lemma 4.1 (iii)] applied with . Let be the directed graph on with edge set , where runs from to . Note that these are distinct, since as has no degenerate elements. Let be the underlying undirected graph, with pairs of directed edges identified into one undirected edge inheriting its orientations from .
Thus, . In fact, let us remember for the definition of also the information from of which orientation of an edge of is and which is . Unlike with arbitrary tree sets, the elements of tree sets of the form thus come with fixed names: those inherited from as defined above.
For every finite tree set , the graph is a tree on .
For every node , the set of its incoming edges is precisely the set of all that are maximal in the orientation of . As is finite, is the down-closure of its maximal elements, so these are splitting stars of . Conversely, every set splitting is clearly of this form: pick , and notice that for by [2, Lemma 4.1 (iii)].
Similarly, the finiteness of implies by [2, Lemma 4.2] that every is the downclosure of its maximal elements, so for every maximal in , giving . Let us now prove that is a tree.
We noted before that for all , so has no loops. In fact, is acyclic. Indeed, if are the edges of an oriented cycle in , then each of these and the inverse of its (cyclic) successor lie in a common set . Since these are stars of separations [2, Lemma 4.5], we have with a contradiction.
To see that is connected, let be nodes in different components, with maximum. Let be maximal in (which we may assume is non-empty). Then is maximal also in : any greater than would also lie in , and hence so would by the consistency of (which also orients ). Replacing in with therefore changes into an orientation of that is again consistent, by the maximality of in . In the separation is maximal: for any we have , so by the consistency of and hence also , giving . Hence , and in particular lies in the same component of as . Since it agrees with on more separations than does, we have a contradiction to the choice of and . ∎
Our aim was to show that every finite tree set is the edge tree set of some tree. In order for Lemma 3.2 to imply this we still need to know that coincides with not only as a set, which it does by definition, but also as a poset: that the is indeed the edge tree set of . Part (i) of the Theorem 3.3 below makes this precise.
Theorem 3.3 (ii) says that the tree whose edge tree set represents , as provided by (i), is unique up to a canonical graph isomorphism: if is the edge tree set also of some other tree , then that tree is isomorphic to by an isomorphism that can be defined just in terms of .
Given a tree , write for the orientation of that orients every edge of towards .
For every finite regular tree set , the identity is a tree set isomorphism between and .
For every finite tree , the map is a graph isomorphism between and .
Note that if (ii) is applied to a tree of the form , then is the identity on , by (i) applied to .
(i) The fact that elements of are inverse to each other also in was built into the definition of .
It remains to show that elements satisfy in if and only if they do so in . Since and are both regular, in either of them implies that , so we may assume this. The equivalence of the two assertions follows by induction on the length of the unique – path in , using the transitivity of , once we have shown it for , that is, when and share a vertex of .
Since by assumption, this means that the consistent orientation of has distinct maximal elements that are orientations of and , respectively. These form a proper star in , because they are both maximal in and is consistent and antisymmetric (and is regular). And they form a proper star in , because they are both oriented towards and incident with . Since, in any separation system, distinct elements of a proper star and their inverses are each related as required by the star property and not in any other way, the orientations of and are related in as they are in .
(ii) For each , the orientation of is clearly consistent, so it is an element of . As and differ on every edge of between and , the map is injective. To see that it is surjective, recall from [2, Lemma 4.2] that the elements of are precisely the down-closures of the subsets splitting , which by Lemma 3.1 are the sets . But the down-closure of is precisely . Thus, our map is a bijection from to .
To see that is a graph isomorphism, notice that for any edge its orientation from to is maximal in , while its other orientation is maximal in . Hence is an edge of between its vertices and .
For the converse note first that, given adjacent vertices and of , the edge joining them in is the only edge of which and orient differently. Indeed, by definition of , the edge has an orientation that is maximal in , and whose inverse is maximal in . By definition of and , this means that is the edge of .
Now if two vertices are not adjacent in , then the – path in contains distinct edges and . As and disagree on both these edges, they cannot be adjacent in . ∎
Infinite tree sets need not be isomorphic to the edge tree set of a tree. Indeed, as discussed in the introduction, one of our motivations for studying tree sets is that they can capture tree-like structures in infinite combinatorics that actual trees cannot represent.
For infinite tree sets that can be represented by a tree , we can still use its splitting stars as the nodes of (Lemma 3.1) and define its edges as earlier. But note that may have consistent orientations that are not the down-closure of a splitting star, and so the nodes of may correspond to only proper a subset of . Indeed, orienting the edges of towards an end of is consistent but such orientations have no maximal elements.
4 Tree sets from order trees
An order tree, for the purpose of this paper, is a poset in which the down set
below every element is a chain. We do not require this chain to be well-ordered. To ensure that order trees induce order trees on the subsets of their ground set, we also do not require that every two elements have a common lower bound. Order trees that do have this property will be called connected.
Order trees are often used to describe the tree-likeness of other combinatorial structures. In such contexts it can be unfortunate that they come with more information than just this tree-likeness, and one has to find ways of ‘forgetting’ the irrelevant additional information.
Theorem 4.4 below offers a way to do this: it canonically splits the information inherent in an order tree into the ‘tree part’ represented by an unoriented tree set, and an ‘orienting part’ represented by an orientation of this tree set. These orientations will be consistent. Indeed, we shall see that order trees correspond precisely to consistently oriented tree sets, finite or infinite.
As in Section 3, let us first define for a given order tree a tree set and a consistent orientation of , and then conversely for a given tree set and any consistent orientation of an order tree . Applying these two operations in turn, starting from either an arbitrary order tree or from an arbitrary consistently oriented tree set, will yield an automorphism of order trees or of tree sets – in fact, the identity or something as close to the identity as is formally possible.
For the first part, let be any order tree. Our aim is to extend to a tree set, i.e., to find a tree set such that is a subposet of . So we have to add some inverses. Let be a set disjoint from and such that whenever . Extend to by letting
if and only if in ;
if and only if are incomparable in .
For every let ; this defines an involution on . Let and .
Whenever is an order tree, is a regular tree set, and is a consistent orientation of .
For a proof that is a regular tree set, the only nontrivial claim to check is that is transitive on .
Consider any . Suppose first that . The first inequality implies, by our definition of , that and are incomparable in . But then so are and (giving as desired): if then , which makes and comparable (which they are not) since is an order tree, while if then in , again contradicting the incomparability of and . Similarly if then , which as just seen implies and hence . This covers all cases that do not follow at once from the transitivity of on .
Finally, is a consistent orientation of , since and we never have for any . ∎
We remark that, if is connected, then is in fact the unique smallest regular tree set to which extends. Indeed, any tree set that induces on a subset must also contain a set of inverses. Let us show that will be disjoint from if is regular. Suppose that are such that (and hence ). If and are comparable, with say, then this makes small, contradicting the regularity of . But if they are incomparable, there will be an below both (since is connected), so our assumption of makes trivial (and hence small), a contradiction.
The proof of our remark will be completed by the following uniqueness lemma, which has the disjointness of and built into its premise and therefore holds also for disconnected order trees. Note that if is connected and is as specified in the lemma, then is necessarily consistent in : otherwise there are are such that , and by the connectedness of there exists with and , so is small, contradicting the regularity of .
Let be a regular tree set. Let be antisymmetric and such that is an order tree. If is consistent in , then induces on .
The involutions in and in agree by definition of .101010Recall that, formally, we did not specify precisely in the definition of : we took ‘any’ set disjoint from and with a bijection from to . The intended reading of Lemma 4.2 is that this is now the set , whose elements are specified as by the involution in . The fact that this agrees with the involution in is then tautological.
It remains to show that and in define the same ordering on . Since and are regular, and are incomparable in both, for all . Now consider distinct . If and are comparable, with say, we must have in by (1), in agreement with our definition of . Assume now that and are incomparable. The consistency of in rules out that . But since is nested, and must have comparable orientations. The only case left is that , as we defined it for . ∎
Lemma 4.1 showed us how to extend, canonically, a given order tree to a regular tree set in which its complement is consistent. We now show that, conversely, deleting a consistent orientation from a regular tree set leaves an order tree.
Given a regular tree set and a consistent orientation of , let be the subposet of induced by .
Whenever is a regular tree set and is a consistent orientation of , the poset is an order tree.
Note first that for we have , since is regular. For our proof that is an order tree, consider with , and let us show that are comparable in . If not, then is comparable with (and with ), because and have comparable orientations since is a tree set. Since is consistent we cannot have , so . But this implies that with a contradiction, since has no small elements. ∎
We have seen that every order tree canonically gives rise to a consistently oriented tree set, and that every consistently oriented tree set canonically induces an order tree. Let us now show that these maps are, essentially, inverse to each other.
When we convert a given order tree into an oriented tree set, , we start by adding a set of inverses disjoint from , and this set will be the desired orientation of . Converting and back into an order tree then just consists of deleting from the poset . This takes us back not only to the original ground set of , but the ordering of on is preserved in the back-and-forth process. Theorem 4.4 (ii) below expresses this.
Going the other way is entails a small technical complication. When we convert a tree set , given with a consistent orientation , into an order tree by deleting from the poset , we cannot hope to get back if we then expand canonially to a tree set , because the canonical process of extending has no knowledge of the actual set we deleted. All we can hope for is that the set we add corresponds to as naturally as possible, and this is what Theorem 4.4 (i) will say. Let us express this formally.
Given a regular tree set and a consistent orientation of let , where , be the identity id on and commute on with the composition whose two maps are the involutions on and on , respectively. Call the canonization of given .
Given a regular tree set and a consistent orientation of , let and . Then the canonization of given is an isomorphism of tree sets that induces the identity on and maps to .
Given an order tree , the identity on is an order isomorphism from to .
(i) As is an orientation of which, being a tree set, has no degenerate elements, is the disjoint union of and . The latter, with its ordering induced by , is an order tree by Lemma 4.3. The assertion now follows from Lemma 4.2.
(ii) By Lemma 4.1, is a regular tree set of which is a consistent orientation. By definition, induces the original ordering of on its ground set . But , by its definition as , also induces this ordering on . Hence and induce the same ordering on their common ground set . ∎
5 Tree sets from nested subsets of a set
Let be a non-empty set. The power set of is a separation system with respect to inclusion and taking complements in . It contains the empty set as a small element, but every nested subset of is a regular tree set.
For compatibility with our earlier notion of set separations, let us refer to subsets of as special kinds of separations of : those of the form . A bipartition of , then, is an ordered pair of disjoint non-empty subsets of whose union is . The bipartitions of form a separation system with respect to their natural ordering defined by and the involution . This separation system has no small elements, so every nested symmetric subset is a regular tree set.
Conversely, every regular tree set can be represented by set bipartitions:
Every regular tree is isomorphic to a tree set of bipartitions of a set.
Given a regular tree set , let and consider
where consists of and the elements of strictly below and their inverses (but not ). The sets and forming a pair in are disjoint because has no trivial elements, and have union all of because is nested. Thus, consists of bipartitions of , and in particular . Clearly, is a bijection from to that commutes with the involutions on and and preserves their orderings. In particular, is a tree set isomorphic to . ∎
The set in Theorem 5.1 is a little larger than necessary. This is best visible when is the edge tree set of a finite tree . Then the elements of correspond to (oriented) bipartitions of the vertices of . So we could represent by these oriented bipartitions of , which has about half as many elements as the set chosen for in our proof of Theorem 5.1.
Section 3 tells us how to generalize this idea to tree sets that are not edge tree sets of a finite tree: the vertices of in the example correspond to the consistent orientations of . So let us try to use these directly to form .
Given , let be the set of consistent orientations of . Every defines a bipartition of : into the set of consistent orientations of containing and the set of those containing . Note that this is indeed a bipartition of ; in particular, and are non-empty by [2, Lemma 4.1 (i)] applied to and , respectively.