One-Shot Capacity of Discrete Channels
Shannon defined channel capacity as the highest rate at which there exists a sequence of codes of block length such that the error probability goes to zero as goes to infinity. In this definition, it is implicit that the block length, which can be viewed as the number of available channel uses, is unlimited. This is not the case when the transmission power must be concentrated on a single transmission, most notably in military scenarios with adversarial conditions or delay-tolerant networks with random short encounters. A natural question arises: how much information can we transmit in a single use of the channel? We give a precise characterization of the one-shot capacity of discrete channels, defined as the maximum number of bits that can be transmitted in a single use of a channel with an error probability that does not exceed a prescribed value. This capacity definition is shown to be useful and significantly different from the zero-error problem statement.
Shannon’s notion of channel capacity  is asymptotic in the sense that the number of channels uses (or, equivalently, the block length of the code) can be arbitrarily large. The rate of a code is defined as the ratio between the number of input symbols and the number of channel uses required to transmit them. A rate is said to be achievable if there exists a sequence of codes (of that rate) with block length , whose error probability goes to zero asymptotically as goes to infinity. The behavior of the channel capacity with a limited number of channel uses is less well understood. A typical approach is to consider the rate at which the error probability decays to zero, which motivates the study of error exponents ,,. Beyond the aforementioned definitions, it is also reasonable to ask what rates can be achieved when the error probability must be precisely zero. In , Shannon assumes once again that the channel is available as many times as necessary and defines the zero-error capacity as the supremum of the independence numbers of the extensions of the confusion graph .
A different question is how much information can we convey in a single use of the channel or, in other words, what is the one-shot capacity of the channel. The question arises for example when the transmission power must be concentrated on a single transmission, most notably in military scenarios with adversarial conditions or delay-tolerant networks with random short encounters. As we have seen, classical definitions of capacity do not encompass these scenarios.
The one-shot capacity problem can also be viewed as a special instance of the single-letter coding problem since, in both problems, the encoder must assign to every source output symbol one channel input symbol (and not a sequence of them). However, to the best of our knowledge, studies on single-letter coding use optimality criteria based on an unlimited number of channel uses. For instance,  characterizes optimal single-letter source-channel codes, with respect to the rate-distortion and capacity-cost functions, which are of asymptotic nature. For comparison, in our study of the one-shot capacity, only a single channel use is considered.
Using a combinatorial approach, the zero-error one-shot capacity of a given channel was considered in . More specifically,  construct a certain undirected graph corresponding to the channel at hand; and characterize the zero-error one-shot capacity by the size of the maximum independent set in .
In this work we generalize the results and combinatorial framework of  to capture communication that allows an error probability below ; namely, we study the -error one-shot capacity. We note that preliminary results on the -error one-shot capacity appear in , which uses smooth min-entropy and a probabilistic approach to develop bounds for the one-shot capacity (also called, single-serving channel capacity). Our work differs from  in that we characterize the exact value of the -error one-shot capacity by means of classical combinatorics.
Our main contributions are as follows:
Problem Formulation: We provide a rigorous mathematical framework for analyzing the -capacity of discrete channels subject to a one-shot constraint. We consider two different metrics of performance: maximum error probability and average error probability.
Operational Interpretation: We illustrate the practical relevance of the one-shot capacity by means of examples where the zero-error one-shot capacity and the -error one-shot capacity present significantly distinct behaviors.
Combinatorial description of the One-Shot Capacity of Discrete Channels: We cast the capacity in terms of the properties of a special graph derived from the channel. For maximum error, we describe the one-shot capacity through the independence number of , whereas for average error we consider the maximum size of sparse sets in .
Complexity Analysis: We show that the problem of computing the one-shot capacity is NP-Hard.
The remainder of the paper is organized as follows. In Section 2, we give a formal definition of the problem at hand, namely the concepts of -maximum and -average one-shot capacity. In Section 3 we present a non-trivial example of a class of channels for which the one-shot capacity is relevant. Our main result for maximum error one-shot capacity is stated and proved in Section 4. In Section 5, we prove that computing the -maximum one-shot capacity is NP-Hard. Finally, in Section 6 we discuss the case of -average one-shot capacity and Section 7 concludes the paper.
We start our problem statement with the usual definition of a discrete channel.
We will refer to such a channel as “the channel described by ″. Next, we present the definition of a one-shot communication scheme over a discrete channel.
We will refer to such a communication scheme as the “ pair”. It is natural to view the set as the set of messages to be transmitted over the channel. Our figure of merit is the probability of error in the decoding process. We consider two different metrics: maximum and average error probability.
We are now ready to define the one-shot capacity of a discrete channel. From an intuitive point of view, we are intrigued by the maximum number of distinct messages (the size of the codebook ) that can be transmitted in a single use of the channel, while ensuring that the error probability (maximum or average) does not exceed a prescribed value . We must first define an admissible pair.
The notion of single-serving capacity is outlined in  as “ the maximum number of bits that can be transmitted in a single use of , such that every symbol can be decoded by an error of at most ″. We formalize this notion by defining the -maximum one-shot capacity as follows:
Similarly, we can define the -average one-shot capacity as follows:
Our goal is to provide a precise characterization of the one-shot capacity.
3Practical Relevance of the One-Shot Capacity
So far, we have formally defined the concept of the -error one-shot capacity. One question that naturally arises is the following: does the -error one-shot capacity significantly differ from the zero-error one-shot capacity, for small values of ? In other words, are there classes of channels for which allowing a small error probability enables the transmission of a significantly larger number of bits than in the zero-error case? In this section, we present a class of channels for which the answer to the previous questions is yes, which asserts for the practical relevance of the -error one-shot capacity notion. Our examples use the maximum error criterion (and thus imply the gap for average error also).
In Fig. , we present an example of a channel in this class. Notice that, given that all symbols are “confusable” (i.e. ), the zero-error one-shot capacity of this channel is zero, . However, by allowing a small error probability, we are able to transmit a significant number of bits.
We start by proving that the -maximum one-shot capacity, , is lower bounded by ( ?). Let , for some (the case is trivial, since by definition ). Consider the codebook and the following decoding function:
For , we have that (where ) and, thus, , because we have that and (because ). With respect to , we have that . Moreover, and . Therefore,
Hence, we have constructed a pair for which and .
Now, we show that is upper bounded by ( ?). Let , for some , and let be a pair for which . Notice that for , we have . Therefore, if , we must have . Thus, since is a function, we have that , which implies that , thus concluding our proof.
The previous example shows that, by allowing for a small probability of error in the decoding process, we are able to transmit a significantly higher number of bits in one use of the channel, in comparison with the case where no errors are allowed. In the case illustrated in Fig. , we have that the -maximum one-shot capacity verifies
4The Case of Maximum Error Probability
In this section, we present a combinatorial description of the one-shot capacity under maximum error . We start by defining the graph that will help us obtain the desired description. For that, we first need to use the following definition which associates with each input symbol a set of output symbols denoted by .
We can view as the set of all possible inverse images of through a decoding function (i.e. all possible ), with verifying . We are now ready to present the definition of the maximum-one-shot graph of the channel described by .
In Fig. , we present the maximum-one-shot graph of the channel in Fig. . Due to the definition of , in the maximum-one-shot graph, nodes represent all the possible (such that is -admissible). To obtain a proper decoding function from the maximum-one-shot graph, we need to find an independent set, since a connection between two nodes represents the incompatibility of two inverse images.
Using these definitions, we are now able to state our main result, which relates the one-shot capacity with the independence number of the previously defined graph.
We prove this theorem by establishing first that one can transmit a codebook of size at least with a single use of the channel. We then show that this is the best one can do.
Let be the maximum-one-shot graph of the channel and let be a maximum independent set in . Let be the set of symbols in that are represented in , i.e.
For each , let be the set of output symbols that are represented in the same node as in , i.e.
Notice that, since is an independent set in and all pairs of nodes of the form and are connected in , we have that is unique and properly defined. Let .
Now, consider the decoder constructed as follows:
for , we set , where is such that ;
for , we set , where is some symbol in .
We have that is an independent set in . Thus, for every , there is only one such that . Therefore, the function is well-defined, i.e. We also have that , which is equivalent to . Let . We have that and, therefore, .
Now, we need to analyze the error probability of the pair previously constructed. Let and let . We have that
Notice that, by the construction of , we have that, for , is a node of and, therefore, a node in . Thus, by the definition of the maximum-one-shot graph , we have that , which is equivalent to Therefore, we have that , and this inequality is not dependent on the choice of . Therefore, we have that , , which is equivalent to Thus, we have constructed a pair such that and , which implies that .
We proved that one can transmit symbols with a single use of the channel. Now, we prove that it is not possible to transmit more than that.
Let be a pair such that and . Let . Since , we have that , where . Thus, and, therefore, for , is a node in the maximum-one-shot graph .
Now, notice that, since is a function with as domain, we have that , . Therefore, the set is an independent set in , which implies that . Since , we have that and, therefore, .
5Complexity of the Computation of the One-Shot Capacity
Up to now we have shown that the -error one-shot capacity can be characterized by the independence number of the graph . Computing the independence number is known to be an NP-Hard problem. However, it may be the case that the graphs we obtain in our reduction are of a simple nature allowing us to find their independence number efficiently. In what follows we show that this is not the case.
We will prove the NP-Hardness of the -maximum one-shot capacity problem by reducing the independent set problem in -regular graphs (a known NP-Hard problem ) to an instance of the -maximum one-shot capacity problem, for . The reduction technique is similar to the one used in .
Consider a -regular graph . We will construct a communication channel driven from this graph as follows: the input alphabet is , the output alphabet is and the transition probability distribution is given by
Notice that is well-defined, since is a -regular graph (each node has degree ) and, therefore, . For each , let . Notice that and .
Now, we shall focus on the -maximum one-shot capacity of the previously constructed channel. Let . Let us now construct the maximum-one-shot graph . The node set is composed of elements of the form such that Two nodes and are connected in if and only if or such that . As , notice that for any node in it holds that .
We now show that =. This suffices to prove our assertion since computing the independence number of is NP-Hard and the -maximum one-shot capacity is equal to the (logarithm of the) independence number of . Namely, we prove that a maximum independent set in corresponds to a maximum independent set in the original -regular graph , and vice-versa.
Let be an independent set in . Consider the set . It holds that . Moreover, for any two nodes and in it holds that , and . This implies that , which in turn implies that and are not connected by an edge. We conclude that is an independent set in .
For the other direction, Let be an independent set in . Consider the set . It holds that . Moreover, for any two nodes and in it holds that . This implies that and are not connected by an edge in . We conclude that is an independent set in .
6The Case of Average Error Probability
In this section, we devote our attention to the -average one-shot capacity (Definition in Section 2).
We are now ready to present the definition of the average-one-shot graph of the channel described by .
The previous definition provides us a tool to describe a relationship between the -average one-shot capacity and sparse sets in the average-one-shot graph.
We are now ready to present our main result related to the -average one-shot capacity.
Let be the average-one-shot graph of the channel and let be a maximum -sparse set in . Let be the set of symbols in that are represented in , i.e. Notice that cannot contain two vertices and (as they share an edge of infinite weight), or two vertices and with (for the same reason).
For each , let . Let ^*. Now, consider the decoder constructed as follows:
for , we set .
for , we set , where is some symbol in .
We have that is an -sparse set in .
This implies that . For the other direction, let be a pair in such that . Let . Clearly, . We now show (very similar to the analysis above) that is -sparse. Namely, first notice that does not contain any infinite weight edges (as is a decoding function). Moreover
We conclude that , which concludes our proof.
Intrigued by the capacity of discrete channels that can be used only once, we elaborated on the -one-shot capacity, defined as the maximum number of bits that can be transmitted with one channel use while assuring that the decoding error probability is not greater than . Based on this definition, we introduced the concept of the -one-shot graph associated with a discrete channel and provided an exact characterization of the -one-shot capacity through combinatorial properties of the -one-shot graph. Using this formulation, we prove that computing the -one-shot capacity (for ) is NP-Hard.
The practical relevance of the concept we present in this paper was discussed through a non-trivial example of a class of discrete channels for which the zero-error capacity is null, but allowing for small error probability enables the transmission of a significant number of bits in a single use of the channel.
This work was partially funded by the Fundação para a Ciência e Tecnologia (FCT, Portuguese Foundation for Science and Technology) under grants SFRH-BD-27273-2006 and PTDC/EIA/71362/2006 (WITS project), and by ISF grant 480/08.
- “A mathematical theory of communication,”
C. E. Shannon, Bell System Technical Journal, vol. 27, pp. 379–423, 1948.
- “Lower bounds to error probability for coding on discrete memoryless channels,”
C. E. Shannon, R. G. Gallager, and E. R. Berlekamp, Information and Control, vol. 10, no. 1, pp. 65–103, 1967.
- “A simple derivation of the coding theorem and some applications,”
R. Gallager, IEEE Transactions on Information Theory, vol. 11, no. 1, pp. 3–18, January 1965.
- Information Theory: Coding Theorems for Discrete Memoryless Systems,
I. Csiszar and J. G. Korner, Academic Press, Inc., Orlando, FL, USA, 1982.
- “The zero error capacity of a noisy channel,”
C. E. Shannon, IRE Transactions on Information Theory, vol. 2, no. 3, pp. 8 – 19, September 1956.
- “On the shannon capacity of a graph,”
L. Lovász, IEEE Transactions on Information Theory, vol. IT-25, pp. 1–7, January 1979.
- “Zero-error information theory,”
J. Korner and A. Orlitsky, IEEE Transactions on Information Theory, vol. 44, no. 6, pp. 2207 – 2229, October 1998.
- “To code, or not to code: Lossy source-channel communication revisited,”
M. Gastpar, B. Rimoldi, and M. Vetterli, IEEE Transactions on Information Theory, vol. 49, no. 5, pp. 1147–1158, May 2003.
- “The single-serving channel capacity,”
R. Renner, S. Wolf, and J. Wullschleger, in IEEE International Symposium on Information Theory, Seattle, WA, July 2006, pp. 1424–1427.
- “Some simplified np-complete graph problems,”
M. R. Garey, D. S. Johnson, and L. Stockmeyer, Theory of Computer Science, vol. 1, pp. 237–267, 1976.