Maximal Closed Set and HalfSpace Separations in Finite Closure Systems
Abstract
We investigate some algorithmic properties of closed set and halfspace separation in abstract closure systems. These two problems generalize various problems arising in different fields of artificial intelligence, including predictive models for networks, reasoning in formal concept analysis, or inductive logic programming. Assuming that the underlying closure system is finite and given by the corresponding closure operator, we show that the halfspace separation problem is NPcomplete. In contrast, for the relaxed problem of maximal closed set separation we give a greedy algorithm using linear number of queries (i.e., closure operator calls) and show that this bound is sharp. For a second direction to overcome the negative result above, we consider Kakutani closure systems and prove that they are algorithmically characterized by the greedy algorithm. As one of the major potential application fields, we then focus on Kakutani closure systems over graphs and generalize a fundamental characterization result based on the Pasch axiom to graph structured partitioning of finite sets. In addition, we give a sufficient condition for Kakutani closure systems over graphs in terms of graph minors. For a second application field, we consider closure systems over finite lattices, present an adaptation of the generic greedy algorithm to this kind of closure systems, and consider two potential applications. In particular, we show that for the special case of subset lattices over finite ground sets, e.g., for formal concept lattices, its query complexity is only logarithmic in the size of the lattice, in contrast to the general algorithm. The second application is concerned with finite subsumption lattices in inductive logic programming. We show that our method for separating two sets of firstorder clauses from each other extends the traditional approach based on least general generalizations of firstorder clauses. Though our primary focus is on the generality of the results obtained, we experimentally demonstrate the practical usefulness of the greedy algorithm on binary classification problems in Kakutani and nonKakutani closure systems.
1 Introduction
The theory of binary separation in by hyperplanes goes back to at least Rosenblatt’s pioneer work on perceptron learning in the late fifties [Rosenblatt/58]. Since then several deep results have been published on this topic including, among others, Vapnik and his coworkers seminal paper on support vector machines [Boser/Guyon/Vapnik/92]. The general problem of binary separation in by hyperplanes can be regarded as follows: Given two finite sets , check whether their convex hulls are disjoint, or not. If not then return the answer “No” indicating that and are not separable by a hyperplane. Otherwise, there exists a hyperplane in such that the convex hull of lies completely in one of the two halfspaces defined by the hyperplane and that of in the other one. The class of an unseen point in is then predicted by that of the training examples in the halfspace it belongs to. The correctness of this generic method for is justified by the result of Kakutani [Kakutani37] that any two disjoint convex sets in are always separable by a hyperplane.
While hyperplane separation in is a wellfounded field, the adaptation of the above idea to other types of data, such as graphs and other relational and algebraic structures has received less attention in artificial intelligence. In contrast, abstract halfspaces over finite domains have intensively been studied among others in geometry and theoretical computer science (see, e.g., [Chepoi1994, ellis1952, Kubis2002, vandeVel1984]). Using the fact that the set of all convex hulls in forms a closure system, the underlying idea of adapting hyperplane separation in to arbitrary finite sets is to consider some semantically meaningful closure system over (see, e.g., [Vel1993] for abstract closure structures). A subset of is then considered as an abstract halfspace, if and its complement both belong to . In this field of research there is a distinguished focus on characterization results of special closure systems, called Kakutani closure systems (see, e.g., [Chepoi1994, Vel1993]). This kind of closure systems satisfy the following property: If the closures of two sets are disjoint then they are halfspace separable in the closure system.
Utilizing the results of other fields [Chepoi1994, ellis1952, Kubis2002, vandeVel1984], in this work we focus on the algorithmic aspects of halfspace separation in abstract closure systems over finite domains (or ground sets).
In all results presented in this paper we assume that the closure systems are given implicitly via the corresponding closure operator. This assumption is justified by the fact that the cardinality of a closure system can be exponential in that of the domain. We regard the closure operator as an oracle (or black box) which returns the closure for any subset of the domain in unit time.
Using these assumptions, we first show that the problem of deciding whether two subsets of the ground set are halfspace separable in the underlying abstract closure system is NPcomplete.
In view of this negative result, we then relax the problem of halfspace separation to maximal closed set
To motivate the general setting considered in this work, we present three problems from different fields of artificial intelligence and show that they all deal with maximal or halfspace separation in abstract closure systems.

(machine learning) Our first example is concerned with predictive learning in graphs. More precisely, suppose all vertices of a graph are colored by one of two colors, say red and blue, but the colors are known only for a subset of the vertices. The task is to predict the unknown color for all uncolored vertices. Clearly, if we have no further information about the problem, then there is no chance to improve the predictive performance of random guessing. Suppose we are provided with the additional knowledge about the fully colored graph that for all pairs of monochromatic vertices, all shortest paths connecting them are also monochromatic. Then, utilizing the folklore result that this kind of “convexity” gives rise to a closure operator (see, e.g., [doi:10.1137/0607049]), the problem above can be regarded as a special case of halfspace separation in the corresponding abstract closure system. If the convexity property holds for one of the two colors only, then the problem becomes a special case of the maximal closed set separation problem, independently whether the color is known, or not.

(formal concept analysis) For the second application example, consider the following problem concerning formal concepts [Gan05]: Given disjoint sets and of concepts, find two concepts and such that generalizes all concepts in , but no concept in and specializes all concepts in , but no concept in , or vice versa. Furthermore, need to be maximal with respect to this property. If there are no such and , then the algorithm is required to return the answer “No”. That is, we are interested in finding two maximal “metaconcepts” (i.e., sets of concepts) separating from . For this problem, one can consider the closure system formed by the set of maximal sublattices of the concept lattice and regard the problem as finding a maximal closed set separation of and in that system.

(inductive logic programming) Our third motivating example deals with generalization and specialization of firstorder clauses (see, e.g., [NieWol97]). More precisely, one of the most common problems in inductive logic programming (ILP) [NieWol97] is defined as follows: Given a set of positive and a set of negative firstorder clauses
^{2} , return a firstorder clause that subsumes (or generalizes) all positive and none of the negative clauses, if such a clause exists; otherwise return “No”. It follows from Plotkin’s seminal results [plotkin1970inductive] that such a clause exists if and only if the least general generalization of the positive clauses does not subsume any of the negative ones. Equivalently, the algorithm is required to return the supremum of the smallest ideal in the subsumption lattice that contains all positive examples if it is disjoint with the set of negative clause; o/w the answer “No”. In contrast to this classical problem setting, we select a finite sublattice of the subsumption lattice spanned by certain specialization and generalization of the input clauses and consider the set system defined by the set of ideals and filters of this lattice. Since it is a closure system, the separation of the two clause sets above can be regarded as another special case of the maximal closed set separation problem. Regarding the solution of the two problems, there are two crucial differences. While the traditional ILP problem above treats the positive and negative examples asymmetrically (i.e., considers the smallest ideal containing the positive examples), in the maximal closed set separation problem the two clause sets are regarded symmetrically (i.e., it allows the solution to consist of a generalization of the negative and a specialization of the positive examples as well). Thus, as we show in Section LABEL:sec:KakutaniLattices, the maximal closed set separation problem can have a solution also for such problem instances where there is no consistent hypothesis according to the traditional ILP problem setting.
For the maximal closed set separation problem above we give a simple efficient greedy algorithm and show that it is optimal w.r.t. the number of closure operator calls in the worstcase. As a second approach to resolve the negative complexity result concerning halfspace separability, we then focus on Kakutani closure systems, i.e., in which two sets are halfspace separable if and only if their closures are disjoint (see, e.g., [Chepoi1994, Vel1993]). We first show that any algorithm deciding whether a closure system is Kakutani or not requires exponentially many closure operator calls in the worstcase. Despite this generic negative result, Kakutani closure systems remain highly interesting because there are various closure systems which are known to be Kakutani. We also prove that the greedy algorithm designed for computing maximal closed set separations provides an algorithmic characterization of Kakutani closure systems. This implies that for these systems the output is always a partitioning of the domain into two halfspaces containing the closures of the input sets if and only if their closures are disjoint.
Regarding potential applications of maximal closed set and halfspace separations, we then turn our attention to closure systems over graphs and lattices. In particular, for graphs we consider the closure operator over vertices induced by shortest paths [doi:10.1137/0607049] and generalize first a fundamental characterization result of this kind of Kakutani closure systems that is based on the Pasch axiom [Chepoi1994] to graph structured partitioning of finite sets. Potential practical applications of this more general result include graph clustering (see, e.g., [Schaeffer/2007]) and graph partitioning (see, e.g., [Buluc_etal/16]). Although the Pasch axiom allows for a polynomial time naive algorithm for deciding whether a closure system over graphs is Kakutani or not, the algorithm is practically infeasible even for small graphs. As a second result concerning graphs, we therefore show that closure systems over graphs induced by shortest paths [doi:10.1137/0607049] are Kakutani if they do not contain the bipartite clique as a minor. We note that the converse of this claim is not true in general. This result, together with Chartrand and Harary’s characterization result of outerplanar graphs [Chartrand/Harary/1967], immediately implies that closures systems over outerplanar graphs and hence, over trees are Kakutani.
As a second application field, we consider maximal closed set and halfspace separation in closure systems defined over finite lattices. Utilizing the algebraic properties of finite lattices, we present an adaptation of the generic greedy algorithm to lattices which calculates a maximal ideal and filter that are disjoint and contain the input sets and , respectively. We show that this adaptation has several algorithmic advantages over the original greedy algorithm. In particular, it utilizes that the current ideal and filter in each iteration can be represented by its supremum and infimum, respectively. Furthermore, their disjointness can be decided by comparing these two elements. For the special case that the elements of the lattice are subsets of some finite ground set (e.g., concept lattices [Gan05]), the number of closure operator calls required by the adapted algorithm is quadratic in the cardinality of the ground set. In this way, an exponential speedup can be obtained over the original greedy algorithm. In addition to these results, we also show that the adapted algorithm preserves the characterization property of the generic greedy algorithm, i.e., it provides an algorithmic characterization of Kakutani closure systems over finite lattices. This characterization result is somewhat orthogonal to that formulated in terms of distributivity (see, e.g., [Kubis2002]).
Besides the positive and negative theoretical results discussed above, we present some illustrative experimental results for binary classification in Kakutani and nonKakutani closure systems obtained by the generic greedy algorithm. For the (binary) target labels we select a halfspace from the underlying closure system at random and create the two sets by generating a random sample from this halfspace and its complement. Note that by construction, the closures of the two training sets are always disjoint. Nevertheless, in case of nonKakutani closure systems the output of the algorithm is not necessarily a halfspace separation. To evaluate the predictive performance, we measure the precision on the output sets which are halfspaces in case of Kakutani and disjoint maximal closed sets in case of nonKakutani closure systems. In addition to precision, for nonKakutani closure systems we measure also the recall. In particular, for Kakutani closure systems, we consider the binary vertex classification in trees. Our results clearly demonstrate that a remarkable predictive accuracy can be obtained. For nonKakutani closure systems we look at finite point sets in . Although in this case the output sets are not necessarily separating halfspaces, we obtained surprisingly high precision and recall values. We emphasize that we deliberately have not exploited any domain specific properties in these experiments, as our primary goal was to study the predictive performance of our general purpose greedy algorithm. Accordingly, we therefore have not compared our results to those of the stateoftheart domain specific algorithms.
The rest of the paper is organized as follows. In Section 2 we collect the necessary notions and fix the notation. Section LABEL:sec:maximalclosed is concerned with the complexity issues of halfspace separation and with the relaxed problem of maximal closed set separation in closure systems. Section LABEL:sec:kakutani is devoted to Kakutani closure systems. In Sections LABEL:sec:kakutanigraphs and LABEL:sec:KakutaniLattices we present applications to closure systems over graphs and lattices, respectively. Our experimental results on synthetic datasets for Kakutani and nonKakutani closure systems are reported in Section LABEL:sec:ExperimentalResults. Finally, in Section LABEL:sec:summary we formulate some interesting problems for further research.
2 Preliminaries
In this section we collect the necessary notions and notation for set and closure systems (see, e.g., [Chepoi1994, Vel1993, Davey/Priestley/2002] for good references on closure systems and separation axioms).
Closure Systems
For a set , denotes the power set of . A set system over a ground set is a pair with ; is a closure system if it fulfills the following properties:

and

for all .
Throughout this paper by closure systems we always mean closure systems over finite ground sets (i.e., ). It is a wellknown fact (see, e.g., [Davey/Priestley/2002]) that any closure system can be defined by a closure operator, i.e., function satisfying the following properties for all :

, (extensivity)

whenever , (monotonicity)

. (idempotency)
For a closure system , the corresponding closure operator is defined by
for all . Conversely, for a closure operator over the corresponding closure system, denoted , is defined by the family of its fixed points, i.e.,
Depending on the context we sometimes omit the underlying closure operator from the notation and denote the closure system simply by . The elements of of a closure system will be referred to as closed or convex sets. This latter terminology is justified by the fact that closed sets generalize several properties of convex hulls in . As a straightforward example, for any finite set , the set system with defined by
(1) 
for all is a closure systems, where denotes the convex hull of in . We will refer to this type of closure systems as closure systems.
Footnotes
 Throughout this work we consistently use the nomenclature “closed sets” by noting that “convex” and “closed” are synonyms by the standard terminology of this field.
 In case of logic programming, clauses are restricted to program clauses, where the body may contain negative literals as well (cf. [Lloyd87]).