Partition Reduction for Lossy Data Compression Problem
We consider the computational aspects of lossy data compression problem, where the compression error is determined by a cover of the data space. We propose an algorithm which reduces the number of partitions needed to find the entropy with respect to the compression error. In particular, we show that, in the case of finite cover, the entropy is attained on some partition. We give an algorithmic construction of such partition.
The basic description of lossy data compression consists of the quantization of the data space into partition and (binary) coding for this partition. Based on the approach of A. Rényi’s [1, 2] and E. C. Posner et al. [3, 4, 5], we have recently presented an idea of the entropy which allows to combine these two steps . The main advantage of our description over classical ones is that we consider general probability spaces without metric. It gives us more freedom to define the error of coding.
In this paper we concentrate on the calculation of the entropy defined in . We propose an algorithm which allows to reduce drastically the computational effort to perform the lossy data coding procedure.
To explain precisely our results let us first introduce basic definitions and give their interpretations. In this paper, if not stated otherwise, we always assume that is a subprobability space111We assume that is a measurable space and .. As it was mentioned, the procedure of lossy data coding consists of the quantization of data space into partition and binary coding for this partition. We say that family is a partition if it is countable family of measurable, pairwise disjoint subsets of such that
During encoding we map every given point to the unique if and only if . Binary coding for the partition can be simply obtained by Huffman coding of elements of .
The statistical amount of information given by optimal lossy coding of by elements of partition is determined by the entropy of which is :
where , for and is the Shannon function.
The coding defined by a given partition causes specific level of error. To control the maximal error, we fix an error-control family which is just a measurable cover of . Then we consider only such partitions which are finner than i.e. we desire that for every there exists such that . If this is the case then we say that is -acceptable and we write .
To understand better the definition of the error-control family let us consider the following example.
Let be a family of all intervals in with length . Every -acceptable partition consists of sets with diameter at most . Then, after encoding determined by such partition, every symbol can be decoded at least with the precision . The above error-control family was considered by A. Rényi [1, 2] in his definition of the entropy dimension. As the natural extensions he also studied the error-control families built by all balls with given radius in general metric spaces. Similar approach was also used by E. C. Posner [3, 4, 5] in his definition of -entropy222E. C. Posner considered in fact -entropy which differs slightly from our approach..
In the case of general measures, it seems to be more natural to vary the lengths of intervals from the error-control family. Less probable events should be coded with lower precision (longer intervals) while more probable ones with higher accuracy (shorter intervals). Our approach allows to deal easily with such situations.
To describe the best lossy coding determined by -acceptable partitions, we define the entropy of as
Let us observe what is the main difficulty in the application of this approach to the lossy data coding:
Let us consider the data space and the error-control family . In such simple situation there exists uncountable number of -acceptable partitions which have to be considered to find .
In this paper, we show how to reduce the aforementioned problem to the at most countable one. In the next section, we propose an algorithm which for a given partition , allows to construct -acceptable partition with not greater entropy than , where denotes the sigma algebra generated by (see Algorithm II.1).
As a consequence we obtain that the entropy can be realized only by partitions (see Corollary III.1). In the case of finite error-control families , we get an algorithmic construction of optimal -acceptable partition. More precisely, if is an error-control family then there exists sets such that (see Corollary III.3):
Ii Algorithm for Partition Reduction
In this section we present an algorithm which for a given -acceptable partition constructs -acceptable partition with not greater entropy. We give the detailed explanation that .
We first establish the notation: for a given family of subsets of and set , we denote:
Let be an error-control family on and let be a -acceptable partition of . We build a family according to the following algorithm: Algorithm II.1.
We are going to show that Algorithm II.1 produces the partition with not greater entropy than . Before that, for the convenience of the reader, we first recall two important facts, which we will refer to in further considerations.
Given numbers and such that , we have:
For the proof we refer the reader to [7, Section 6] where similar problem is illustrated for . ∎
Consequence of Lebesgue Theorem (see ) Let be summable i.e. and be a sequence of functions such that , for . If is pointwise convergent, for every , then is summable and
Let us move to the analysis of Algorithm II.1. We first check what happens in the single iteration of the algorithm.
We consider an error-control family and a -acceptable partition of . Let be such that:
If is chosen so that then
Clearly, if then the inequality (9) holds trivially. Thus we assume that .
Let us observe that it is enough to consider only elements of with non zero measure – the number of such sets can be at most countable. Thus, let us assume that (the case when is finite can be treated in similar manner).
For simplicity we put . For every , we consider the sequence of sets, defined by
Clearly, for , we have
To complete the proof it is sufficient to derive that for every , we have:
Making use of Observation II.1, we obtain
which proves (16).
To calculate , we will use the Consequence of Lebesgue Theorem. We consider a sequence of functions
Let us observe that the Shannon function is increasing on and decreasing on . Thus for a certain ,
for every . Since then
for every .
As the sequence of functions satisfies the assumptions of the Consequence of Lebesgue Theorem then, we get
Consequently, we have
which completes the proof. ∎
We are ready to summarize the analysis of Algorithm II.1. We present it in the following two theorems.
Let be an error-control family on and let be a -acceptable partition of . Family constructed by the Algorithm II.1 is a partition of .
Directly from the Algorithm II.1, we get that is countable family of pairwise disjoint sets.
Let us assume that , since the case when is finite family is straightforward. To prove that
we will use the Consequence of Lebesgue Theorem.
For every , we define a function by
To see that the sequence is pointwise convergent, we apply the indirect reasoning. Let and let be such that, for every ,
We put . We assume that we have already chosen sets . Since then , for every . Hence, we have
as is a family of pairwise disjoint sets. Consequently,
which is the contradiction. The sequence is convergent.
Finally, making use of Lebesgue Theorem, we obtain
Let be an error-control family on and let be a -acceptable partition of . Partition built by Algorithm II.1 satisfies:
If then the inequality (46) is straightforward. Thus let us discuss the case when .
We denote , since at most countable number of elements of partition can have positive measure (the case when is finite follows similarly). We will use the notation introduced in Algorithm II.1.
Directly from Lemma II.1, we obtain
Consequently, for every , we get
Our goal is to show that
for every .
Making use of (48), we have
for every .
We will calculate using the Consequence of Lebesgue Theorem for a sequence of functions , defined by
Similarly to the proof of Lemma II.1, we may assume that there exists such that
for every . Moreover,
for every since is a partition of .
Making use of the Consequence of Lebesgue Theorem, we get
Consequently, for every , we have
which completes the proof. ∎
Iii Concluding Remarks
We have seen that in computing the entropy with respect to the error-control family it is sufficient to consider only partitions constructed from the sigma algebra generated by . Thus, we may rewritten the definition of the entropy with respect to :
Let us observe that Algorithm II.1 shows how to find a -acceptable partition with the entropy arbitrarily close to :
Let be an error-control family of . For any number , there exists partition such that
For simplicity let us assume that (the case when is finite or uncountable follows in similar way). Then the partition , which satisfies the assertion, is of the form:
for specific permutation of natural numbers. ∎
When is a finite family then the entropy of is always attained on some partition . More precisely, we have:
Let be element error-control family, where . Then there exist sets , for specific , such that
To see that the entropy with respect to arbitrary, possibly infinite, error-control family does not have to be attained on any partition, we use trivial example from [6, Example II.1]:
Let us consider the open segment with sigma algebra generated by all Borel subsets of , Lebesgue measure and error control family, defined by
One can verify that but clearly , for every -acceptable partition .
As an open problem we leave the following question:
Let be an error-control family. We assume that if there exists such that , for every , then also . We ask if the entropy with respect to is realized by some -acceptable partition .
-  A. Rényi, “On measures of entropy an information,” Proc. Fourth Berkeley Symp. on Math. Statist.and Prob., vol. 1, pp. 647–561, 1961.
-  ——, “On the dimension and entropy of probability distributions,” Acta Mathematica Hungarica, vol. 10, no. 1–2, pp. 193–215, 1959.
-  E. C. Posner, E. R. Rodemich, and H. Rumsey, “Epsilon entropy of stochastic processes,” The Annals of Mathematical Statistics, vol. 38, pp. 1000–1020, 1967.
-  E. C. Posner and E. R. Rodemich, “Epsilon entropy and data compression,” The Annals of Mathematical Statistics, vol. 42, pp. 2079–2125, 1971.
-  ——, “Epsilon entropy of stochastic processes with continuous paths,” The Annals of Probability, vol. 1, no. 4, pp. 674–689, 1973.
-  M. Śmieja and J. Tabor, “Entropy of the mixture of sources and entropy dimension,” to appear in IEEE Transactions on Information Theory, vol. 58, no. 5, 2012.
-  C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, pp. 379–423, 623–656, 1948.
-  J. F. C. Kingman and S. J. Taylor, Introduction to measures and probability. Cambridge University Press, 1966.