Maximal Closed Set and Half-Space Separations in Finite Closure Systems
We investigate some algorithmic properties of closed set and half-space separation in abstract closure systems. These two problems generalize various problems arising in different fields of artificial intelligence, including predictive models for networks, reasoning in formal concept analysis, or inductive logic programming. Assuming that the underlying closure system is finite and given by the corresponding closure operator, we show that the half-space separation problem is NP-complete. In contrast, for the relaxed problem of maximal closed set separation we give a greedy algorithm using linear number of queries (i.e., closure operator calls) and show that this bound is sharp. For a second direction to overcome the negative result above, we consider Kakutani closure systems and prove that they are algorithmically characterized by the greedy algorithm. As one of the major potential application fields, we then focus on Kakutani closure systems over graphs and generalize a fundamental characterization result based on the Pasch axiom to graph structured partitioning of finite sets. In addition, we give a sufficient condition for Kakutani closure systems over graphs in terms of graph minors. For a second application field, we consider closure systems over finite lattices, present an adaptation of the generic greedy algorithm to this kind of closure systems, and consider two potential applications. In particular, we show that for the special case of subset lattices over finite ground sets, e.g., for formal concept lattices, its query complexity is only logarithmic in the size of the lattice, in contrast to the general algorithm. The second application is concerned with finite subsumption lattices in inductive logic programming. We show that our method for separating two sets of first-order clauses from each other extends the traditional approach based on least general generalizations of first-order clauses. Though our primary focus is on the generality of the results obtained, we experimentally demonstrate the practical usefulness of the greedy algorithm on binary classification problems in Kakutani and non-Kakutani closure systems.
The theory of binary separation in by hyperplanes goes back to at least Rosenblatt’s pioneer work on perceptron learning in the late fifties [Rosenblatt/58]. Since then several deep results have been published on this topic including, among others, Vapnik and his co-workers seminal paper on support vector machines [Boser/Guyon/Vapnik/92]. The general problem of binary separation in by hyperplanes can be regarded as follows: Given two finite sets , check whether their convex hulls are disjoint, or not. If not then return the answer “No” indicating that and are not separable by a hyperplane. Otherwise, there exists a hyperplane in such that the convex hull of lies completely in one of the two half-spaces defined by the hyperplane and that of in the other one. The class of an unseen point in is then predicted by that of the training examples in the half-space it belongs to. The correctness of this generic method for is justified by the result of Kakutani [Kakutani37] that any two disjoint convex sets in are always separable by a hyperplane.
While hyperplane separation in is a well-founded field, the adaptation of the above idea to other types of data, such as graphs and other relational and algebraic structures has received less attention in artificial intelligence. In contrast, abstract half-spaces over finite domains have intensively been studied among others in geometry and theoretical computer science (see, e.g., [Chepoi1994, ellis1952, Kubis2002, vandeVel1984]). Using the fact that the set of all convex hulls in forms a closure system, the underlying idea of adapting hyperplane separation in to arbitrary finite sets is to consider some semantically meaningful closure system over (see, e.g., [Vel1993] for abstract closure structures). A subset of is then considered as an abstract half-space, if and its complement both belong to . In this field of research there is a distinguished focus on characterization results of special closure systems, called Kakutani closure systems (see, e.g., [Chepoi1994, Vel1993]). This kind of closure systems satisfy the following property: If the closures of two sets are disjoint then they are half-space separable in the closure system.
Utilizing the results of other fields [Chepoi1994, ellis1952, Kubis2002, vandeVel1984], in this work we focus on the algorithmic aspects of half-space separation in abstract closure systems over finite domains (or ground sets).
In all results presented in this paper we assume that the closure systems are given implicitly via the corresponding closure operator. This assumption is justified by the fact that the cardinality of a closure system can be exponential in that of the domain. We regard the closure operator as an oracle (or black box) which returns the closure for any subset of the domain in unit time.
Using these assumptions, we first show that the problem of deciding whether two subsets of the ground set are half-space separable in the underlying abstract closure system is NP-complete.
In view of this negative result, we then relax the problem of half-space separation to maximal closed set
To motivate the general setting considered in this work, we present three problems from different fields of artificial intelligence and show that they all deal with maximal or half-space separation in abstract closure systems.
(machine learning) Our first example is concerned with predictive learning in graphs. More precisely, suppose all vertices of a graph are colored by one of two colors, say red and blue, but the colors are known only for a subset of the vertices. The task is to predict the unknown color for all uncolored vertices. Clearly, if we have no further information about the problem, then there is no chance to improve the predictive performance of random guessing. Suppose we are provided with the additional knowledge about the fully colored graph that for all pairs of monochromatic vertices, all shortest paths connecting them are also monochromatic. Then, utilizing the folklore result that this kind of “convexity” gives rise to a closure operator (see, e.g., [doi:10.1137/0607049]), the problem above can be regarded as a special case of half-space separation in the corresponding abstract closure system. If the convexity property holds for one of the two colors only, then the problem becomes a special case of the maximal closed set separation problem, independently whether the color is known, or not.
(formal concept analysis) For the second application example, consider the following problem concerning formal concepts [Gan05]: Given disjoint sets and of concepts, find two concepts and such that generalizes all concepts in , but no concept in and specializes all concepts in , but no concept in , or vice versa. Furthermore, need to be maximal with respect to this property. If there are no such and , then the algorithm is required to return the answer “No”. That is, we are interested in finding two maximal “meta-concepts” (i.e., sets of concepts) separating from . For this problem, one can consider the closure system formed by the set of maximal sublattices of the concept lattice and regard the problem as finding a maximal closed set separation of and in that system.
(inductive logic programming) Our third motivating example deals with generalization and specialization of first-order clauses (see, e.g., [NieWol97]). More precisely, one of the most common problems in inductive logic programming (ILP) [NieWol97] is defined as follows: Given a set of positive and a set of negative first-order clauses
2, return a first-order clause that subsumes (or generalizes) all positive and none of the negative clauses, if such a clause exists; otherwise return “No”. It follows from Plotkin’s seminal results [plotkin1970inductive] that such a clause exists if and only if the least general generalization of the positive clauses does not subsume any of the negative ones. Equivalently, the algorithm is required to return the supremum of the smallest ideal in the subsumption lattice that contains all positive examples if it is disjoint with the set of negative clause; o/w the answer “No”. In contrast to this classical problem setting, we select a finite sublattice of the subsumption lattice spanned by certain specialization and generalization of the input clauses and consider the set system defined by the set of ideals and filters of this lattice. Since it is a closure system, the separation of the two clause sets above can be regarded as another special case of the maximal closed set separation problem. Regarding the solution of the two problems, there are two crucial differences. While the traditional ILP problem above treats the positive and negative examples asymmetrically (i.e., considers the smallest ideal containing the positive examples), in the maximal closed set separation problem the two clause sets are regarded symmetrically (i.e., it allows the solution to consist of a generalization of the negative and a specialization of the positive examples as well). Thus, as we show in Section LABEL:sec:KakutaniLattices, the maximal closed set separation problem can have a solution also for such problem instances where there is no consistent hypothesis according to the traditional ILP problem setting.
For the maximal closed set separation problem above we give a simple efficient greedy algorithm and show that it is optimal w.r.t. the number of closure operator calls in the worst-case. As a second approach to resolve the negative complexity result concerning half-space separability, we then focus on Kakutani closure systems, i.e., in which two sets are half-space separable if and only if their closures are disjoint (see, e.g., [Chepoi1994, Vel1993]). We first show that any algorithm deciding whether a closure system is Kakutani or not requires exponentially many closure operator calls in the worst-case. Despite this generic negative result, Kakutani closure systems remain highly interesting because there are various closure systems which are known to be Kakutani. We also prove that the greedy algorithm designed for computing maximal closed set separations provides an algorithmic characterization of Kakutani closure systems. This implies that for these systems the output is always a partitioning of the domain into two half-spaces containing the closures of the input sets if and only if their closures are disjoint.
Regarding potential applications of maximal closed set and half-space separations, we then turn our attention to closure systems over graphs and lattices. In particular, for graphs we consider the closure operator over vertices induced by shortest paths [doi:10.1137/0607049] and generalize first a fundamental characterization result of this kind of Kakutani closure systems that is based on the Pasch axiom [Chepoi1994] to graph structured partitioning of finite sets. Potential practical applications of this more general result include graph clustering (see, e.g., [Schaeffer/2007]) and graph partitioning (see, e.g., [Buluc_etal/16]). Although the Pasch axiom allows for a polynomial time naive algorithm for deciding whether a closure system over graphs is Kakutani or not, the algorithm is practically infeasible even for small graphs. As a second result concerning graphs, we therefore show that closure systems over graphs induced by shortest paths [doi:10.1137/0607049] are Kakutani if they do not contain the bipartite clique as a minor. We note that the converse of this claim is not true in general. This result, together with Chartrand and Harary’s characterization result of outerplanar graphs [Chartrand/Harary/1967], immediately implies that closures systems over outerplanar graphs and hence, over trees are Kakutani.
As a second application field, we consider maximal closed set and half-space separation in closure systems defined over finite lattices. Utilizing the algebraic properties of finite lattices, we present an adaptation of the generic greedy algorithm to lattices which calculates a maximal ideal and filter that are disjoint and contain the input sets and , respectively. We show that this adaptation has several algorithmic advantages over the original greedy algorithm. In particular, it utilizes that the current ideal and filter in each iteration can be represented by its supremum and infimum, respectively. Furthermore, their disjointness can be decided by comparing these two elements. For the special case that the elements of the lattice are subsets of some finite ground set (e.g., concept lattices [Gan05]), the number of closure operator calls required by the adapted algorithm is quadratic in the cardinality of the ground set. In this way, an exponential speed-up can be obtained over the original greedy algorithm. In addition to these results, we also show that the adapted algorithm preserves the characterization property of the generic greedy algorithm, i.e., it provides an algorithmic characterization of Kakutani closure systems over finite lattices. This characterization result is somewhat orthogonal to that formulated in terms of distributivity (see, e.g., [Kubis2002]).
Besides the positive and negative theoretical results discussed above, we present some illustrative experimental results for binary classification in Kakutani and non-Kakutani closure systems obtained by the generic greedy algorithm. For the (binary) target labels we select a half-space from the underlying closure system at random and create the two sets by generating a random sample from this half-space and its complement. Note that by construction, the closures of the two training sets are always disjoint. Nevertheless, in case of non-Kakutani closure systems the output of the algorithm is not necessarily a half-space separation. To evaluate the predictive performance, we measure the precision on the output sets which are half-spaces in case of Kakutani and disjoint maximal closed sets in case of non-Kakutani closure systems. In addition to precision, for non-Kakutani closure systems we measure also the recall. In particular, for Kakutani closure systems, we consider the binary vertex classification in trees. Our results clearly demonstrate that a remarkable predictive accuracy can be obtained. For non-Kakutani closure systems we look at finite point sets in . Although in this case the output sets are not necessarily separating half-spaces, we obtained surprisingly high precision and recall values. We emphasize that we deliberately have not exploited any domain specific properties in these experiments, as our primary goal was to study the predictive performance of our general purpose greedy algorithm. Accordingly, we therefore have not compared our results to those of the state-of-the-art domain specific algorithms.
The rest of the paper is organized as follows. In Section 2 we collect the necessary notions and fix the notation. Section LABEL:sec:maximalclosed is concerned with the complexity issues of half-space separation and with the relaxed problem of maximal closed set separation in closure systems. Section LABEL:sec:kakutani is devoted to Kakutani closure systems. In Sections LABEL:sec:kakutanigraphs and LABEL:sec:KakutaniLattices we present applications to closure systems over graphs and lattices, respectively. Our experimental results on synthetic datasets for Kakutani and non-Kakutani closure systems are reported in Section LABEL:sec:ExperimentalResults. Finally, in Section LABEL:sec:summary we formulate some interesting problems for further research.
In this section we collect the necessary notions and notation for set and closure systems (see, e.g., [Chepoi1994, Vel1993, Davey/Priestley/2002] for good references on closure systems and separation axioms).
For a set , denotes the power set of . A set system over a ground set is a pair with ; is a closure system if it fulfills the following properties:
for all .
Throughout this paper by closure systems we always mean closure systems over finite ground sets (i.e., ). It is a well-known fact (see, e.g., [Davey/Priestley/2002]) that any closure system can be defined by a closure operator, i.e., function satisfying the following properties for all :
whenever , (monotonicity)
For a closure system , the corresponding closure operator is defined by
for all . Conversely, for a closure operator over the corresponding closure system, denoted , is defined by the family of its fixed points, i.e.,
Depending on the context we sometimes omit the underlying closure operator from the notation and denote the closure system simply by . The elements of of a closure system will be referred to as closed or convex sets. This latter terminology is justified by the fact that closed sets generalize several properties of convex hulls in . As a straightforward example, for any finite set , the set system with defined by
for all is a closure systems, where denotes the convex hull of in . We will refer to this type of closure systems as -closure systems.
- Throughout this work we consistently use the nomenclature “closed sets” by noting that “convex” and “closed” are synonyms by the standard terminology of this field.
- In case of logic programming, clauses are restricted to program clauses, where the body may contain negative literals as well (cf. [Lloyd87]).