Unlabeled Compression Schemes Exceeding the VC-dimension
In this note we disprove a conjecture of Kuzmin and Warmuth claiming that every family whose VC-dimension is at most admits an unlabeled compression scheme to a sample of size at most . We also study the unlabeled compression schemes of the joins of some families and conjecture that these give a larger gap between the VC-dimension and the size of the smallest unlabeled compression scheme for them.
Terminology: if is a subset of the domain of a function , then we call the restriction the trace of on and we also call an extension of .
Consider a finite set , and fix a family of functions . For and we call the trace a partial function of the family . These are studied extensively in learning theory, where our goal is to reconstruct from some part of it.
Definition 1 (Littlestone and Warmuth ).
A (labeled) compression scheme for is a pair of operations such that
takes a partial function of as an input (called a labeled sample) and returns a trace of ,
takes the output of as input and returns an arbitrary function ,
is an extension of for any partial function of .
That is, instead of , it is enough to store so that we can fully recover the value of over . The size of the compression scheme is the maximum size of the domain of . We denote by the minimum size of a compression scheme for .
Notice that it is not required to be able to reconstruct from .
is not required to be from .
Definition 4 (Vapnik-Chervonenkis ).
Let be a family of functions . We say that shatters if every function has an extension in . The VC-dimension of , , is defined as the size of the largest that is shattered by .
Littlestone and Warmuth  observed that always holds but could not give any compression scheme for general families whose size depended only on . Floyd and Warmuth  conjectured that always holds. (There are simple examples that show that this would be sharp.) Warmuth  even offered $600 reward for a proof that a compression scheme of size always exists, but this has been proved only in special cases.\@xfootnoteFloyd and Warmuth  claimed to have proved it for families of VC-dimension whose size is , i.e., the maximum size allowed by the Sauer-Shelah lemma, but recently an error was discovered in their argument.
In 2015, Moran and Yehudayoff  have managed to prove that a compression scheme exists whose size depends only on , but their bound is exponential in .
Definition 5 (Kuzmin and Warmuth ).
An unlabeled compression scheme for is a pair of operations such that
takes a partial function with domain (called a labeled sample) and returns a (called the compressed sample), which is a subset of ,
takes the output of as input and returns an arbitrary function ,
is an extension of for any partial function of .
That is, unlike in the case of labeled compression schemes, we do not store the value of on the compressed sample, but only some selected sample points. The size of the unlabeled compression scheme is the maximum size of for any partial function . We denote by the minimum size of an unlabeled compression scheme for . Note that trivially holds.
Kuzmin and Warmuth  have proved that and conjectured that equality might hold for every family (a strengthening of the earlier conjecture of Floyd and Warmuth).\@xfootnoteSimilarly to the labeled case, they also made a claim about maximum size families, which seems to contain the same error.
We disprove this last conjecture in a very weak sense; we exhibit a small family for which but . We also discuss possible ways to amplify this gap, but at the moment we do not know any family with for which . (Although a computer search could possibly find such a family - we exhibit some likely candidates.)
2 Lower bound for
Here we define the family for which , and prove these equalities. The base set of is five elements and ; see Figure 1. We think of the base set of as the vertices of a regular pentagon. A 0-1 function on this base set belongs to if and only if it takes the values 1-0-0-1 on some four consecutive vertices.
As we have later found out, this is known in the learning theory literature as ‘Warmuth’s example.’ He constructed it as a simple example of a containment maximal family with that does not reach the maximal size of such a family given by the Sauer-Shelah lemma, which in this case would be .
We will use the property that for any subset of size there are possibilities for the trace for . If consists of three consecutive vertices, then cannot be constant , while if consists of three non-consecutive vertices the constant trace is not possible. Note that this implies that shatters no three element set but it shatters all two element sets, so its VC-dimension is .
We identify the base set of with the residue classes modulo 5, with the neighbors of the vertex being and .
It is easy to construct an unlabeled compression scheme of size : can keep the sample points where the value of the function is , and the reconstruction function returns at every place contained in the compressed sample, and everywhere else. Thus, we only need to prove that .
Suppose by contradiction that there is an unlabeled compression scheme of size two. Let be a size subset of the domain. As we noted above, there are exactly partial functions of . Clearly, must be a distinct proper subset of for each. As there are such subsets, we must have a - correspondence here. In particular, for all , the must be distinct partial functions of .
Let be the set of three consecutive positions in the domain and . Let be the constant partial function defined on and . Here is a partial function of extending , so it must be on and on . Now must be another partial function of , therefore cannot be constant . A symmetric argument shows that if is the set of three non-consecutive positions and , then is not constant .
The observations above imply that . Indeed, if , then applying the observation in the previous paragraph for we obtain and considering we obtain , but this contradicts our observation about . A similar argument shows as well as . The only remaining value, namely therefore completely determines .
Suppose holds for at least three different values of ; then it must hold for two consecutive values, say and . This completely determines and and these functions coincide on contradicting our observation that for distinct proper subsets of , the must also be distinct.
Alternatively we must have for at least three different values of . Then it also holds for two non-consecutive values, say and . This completely determines and and these functions coincide on , a contradiction again. The contradictions prove the theorem. ∎
Note that in the above proof we have in fact showed that there is no compression scheme already in the case when the sample consists of at most values.
3 Upper bounds for ’s
In this section we sketch some upper bounds, i.e., give unlabeled compression schemes for certain families. When we receive a sample , we interpret it as receiving a collection of 0’s and 1’s, and we interpret the compression as keeping some of them (though we only keep the locations, not the values). In the case of , when we receive a sample that contains identical values, then we call them a triple 0 or a triple 1, depending on the value. Recall that a triple 1 can only occur at 3 consecutive positions, and a triple 0 can only occur at 3 non-consecutive positions, so the set of positions determines whether it is a triple 0 or a triple 1.
The join of two families of functions is a family over the disjoiont union of there base sets where if belongs to the base set of and if belongs to the base set of . When we take the join of several copies of the same family, we use the notation
We obviously have , but for compression schemes only follows from the definition, and equality does not always hold, as the following statement shows. Recall that by Theorem 6.
|no triples||keep all 1’s||kept to 1, rest 0|
|triple 1 in||keep triple and 1’s in||triple from position,|
|triple 0 in||keep triple and 0’s in||kept in same|
|triple 1 in||keep triple and 0’s in||triple from position,|
|triple 0 in||keep triple and 1’s in||kept in opposite|
For the proof we need to give an unlabeled compression scheme . There are several possible schemes, one is sketched in Table 1. We write and for the base sets of the two copies of . The compression depends on whether there are, and what type of triples in the labeled sample restricted to the base sets of the two copies of . We denote these base sets by and .
If neither of them contains a triple, we just keep the 1’s in the labeled sample.
If contains a triple 1, but does not contain a triple 1, then we still just keep the 1’s.
If contains a triple 0, but does not contain a triple 0, then we keep all the 0’s in the labeled sample.
If contains a triple 1, but does not contain a triple 0, then keep the triple 1 from , and the 0’s from .
If contains a triple 0, but does not contain a triple 1, then keep the triple 0 from , and the 1’s from .
Note that if the compressed sample contains three positions from either or , then those positions formed a triple in the labeled sample and it was a triple 1 in case of three consecutive positions and a triple 0 in case of three non-consecutive positions. This means that the compressed sample determines which one of the five rules was used to obtain it and the decoding can be constructed accordingly.
Finally, notice that exactly one of the 5 above cases happens for every sample. (Although note that for us it would be sufficient if at least one of them happened for every sample.) ∎
This raises the question of how behaves when . We can prove neither any lower bound that would be better than for any at all (notice that Proposition 9 only provides an upper bound, but we do not know whether in general holds or not), nor show that for every . We make the following conjecture.
exists and is strictly larger than .
We can prove that for . Since the compression schemes are based on similar ideas, we only sketch the scheme for .
|no triple 1||keep all 1’s|
|triple 1 in some|
|but no triple 0 anywhere||keep triple 1 in|
|and 0’s in other ’s|
|exactly one triple 0||keep 0’s|
|exactly one triple 1|
|and least two triple 0’s||fix two triple 0’s and one triple 1; keep non-central triples and central element of central triple, and 1’s from rest|
|least two triple 1’s and least two triple 0’s, and fifth does not have exactly one 1||keep triple 1’s and central elements of triple 0’s, and 1’s from fifth|
|two triple 1’s and least two triple 0’s, and fifth has exactly one 1||keep triple 1’s and non-central elements of triple 0’s, and 1 from fifth|
We denote the copies of ’s by , with indexing .
Among any three positions in a single there is a unique “central” element: the one that is equidistant from the other two elements. We use that the two non-central elements determine the central element uniquely. Although the central element is not enough to determine the other two elements, it becomes enough once we know whether they are the positions in a triple 0 or a triple 1.
Similarly, among any three distinct sets , and , there is a unique central one, whose index is equidistant (modulo 5) from the other two indices. E.g., from , and the central one is , while from , and the central one is . We use again that the non-central copies determine the central one uniquely.
The compression algorithm is sketched in Table 2. This Table needs to be interpreted in a similar fashion as Table 1, this time we omit the lengthy description of the case analysis. Note that for some labeled samples there are more rules to choose from for the compression – in this case, we pick arbitrarily. It is important, however that there is always at least one rule that applies.
We have also omitted the decompression rules, as the compressed sample always determines which rule was used to obtain it. To prove this statement, notice that we only keep three position of the same if they form a triple in the labeled sample. If the first rule is used, no triple is kept. In case the second or third rule is used, a single triple 1 or triple 0 is kept, respectively. If the fourth rule is used, then two triples are kept, not both triple 1’s. Finally if either of the last two rules are used, then at least two triple 1’s are kept. The compressed sample produced by the last two rules are distinguished by the number of elements kept in the sets : if it is in some order, then the last rule was used, otherwise the fifth rule. Once we know which rule produced the compressed sample the decoding can be done accordingly. ∎
4 Further results
In this section we mention some further results. We start by defining some further families.
is obtained from by deleting one function. Because of the symmetry, it does not matter which one, so we delete the function 0-1-1-1-0. Here we represent functions by the sequence of their values on 0, 1, 2, 3, 4. In this family, still any two positions can take any values ( possibilities each), but for some triples we have only possibilities (instead of ).
is the restriction of to four elements of the base set. Again, by symmetry it does not matter which four, so we delete the central element 2. This is useful, because this way also becomes a restriction of .
The lower bounds follow from . For the upper bound, we need to give a compression scheme of size two for . A possible algorithm is sketched in Table 3. Here we list the decoding of compressed samples only. We maintain a symmetry for the reflection to the central element: If the compressed sample is obtained from another compressed sample by reflection, then the decoding is also obtained from the same way. Accordingly, we only list one of and in the Table. We omit the lengthy case analysis of why this compression scheme works. ∎
Now we continue by definining two more families.
is the family of all boolean functions on a base set of elements. Notice that . As shatters its entire base set, we have . We also have as and is shown by the simple unlabeled compression scheme that keeps the 1’s in the labeled sample. On the other hand, can be smaller, e.g., .
is a symmetrizing extension of , with the same number of functions, but one more base element. One can obtain it from by adding an extra element to the base and extending each function in the family to the new element such that the function has three zeros and three ones. Figure 2 depict two functions of . The other eight functions are the rotations of these two. In the family the extra element plays no special role, in fact, is two-transitive, i.e., any pair of elements of its base set can be mapped to any other pair of elements with an automorphism. If we convert the functions of to -element sets, we get the unique design. Since is an extension of , and – it is easy to check that we have equality in both cases, i.e., and .
Some further non-trivial upper bounds can be obtained for the joins involving these families.
|extra is not 1||keep 1’s of||kept 1, others 0|
|extra is 1 and triple 0||keep triple 0||kept 0, others 1|
|extra is 1, no triple 0||keep extra and 0’s||extra 1, rest of kept 0, others 1|
The compression algorithm is sketched in Table 4, with ‘extra’ denoting the only bit of the base set of . ∎
Note that is obtained from by restricting the base set and such a restriction cannot increase the value of , so this also implies . From this we can easily get another proof for as follows. We have , thus , using Proposition 12.
This compression goes similarly to the one presented in Table 1 for . In fact, we can use exactly the same compression scheme unless we get two triples in both ’s, i.e., a labeled sample that contains all elements of the base. There are possibilities for such a sample, and for each we can pick a compression that keeps at least elements from at least one of the two copies of , as these were not used yet. There are such possible compressed samples, we can use a distinct one for each of the problematic labeled samples. This makes the decoding possible. ∎
We end by a summary of the most important questions left open.
Summary of main open questions
How does behave? Does exist?
Is there a for every such that ?
Remarks and acknowledgment
We would like to thank Tamás Mészáros, Shay Moran and Manfred Warmuth for useful discussions and calling our attention to new developments.
-  S. Floyd and M. K. Warmuth, Sample compression, learnability, and the Vapnik-Chervonenkis dimension, in Machine Learning, 21(3):269–304, 1995.
-  D. Kuzmin and M. K. Warmuth, Unlabeled Compression Schemes for Maximum Classes, in Proceedings of the 18th Annual Conference on Computational Learning Theory (COLT 05), Bertinoro, Italy, pp. 591–605, June 2005.
-  N. Littlestone and M. K. Warmuth, Relating data compression and learnability. Unpublished manuscript, obtainable at http://www.cse.ucsc.edu/~manfred, June 10 1986.
-  S. Moran and A. Yehudayoff, Sample compression schemes for VC classes, to appear in the Journal of the ACM.
-  V. N. Vapnik, A. Ya. Chervonenkis, On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities, Theory of Probability & Its Applications. 16(2):264-280, 1971.
-  M. K. Warmuth, Compressing to VC dimension many points, in Proceedings of the 16th Annual Conference on Learning Theory (COLT 03), Washington D.C., USA, August 2003. Springer. Open problem. https://users.soe.ucsc.edu/~manfred/pubs/open/P1.pdf.