Achievable Rates for Noisy Channels with Synchronization Errors1footnote 11footnote 1This research is funded by the National Science Foundation under contract NSF-TF 0830611.2footnote 22footnote 2Part of this work is submitted to 2012 IEEE International Symposium on Information Theory (ISIT).

Achievable Rates for Noisy Channels with Synchronization Errors12

Abstract

We develop several lower bounds on the capacity of binary input symmetric output channels with synchronization errors which also suffer from other types of impairments such as substitutions, erasures, additive white Gaussian noise (AWGN) etc. More precisely, we show that if the channel with synchronization errors can be decomposed into a cascade of two channels where only the first one suffers from synchronization errors and the second one is a memoryless channel, a lower bound on the capacity of the original channel in terms of the capacity of the synchronization error-only channel can be derived. To accomplish this, we derive lower bounds on the mutual information rate between the transmitted and received sequences (for the original channel) for an arbitrary input distribution, and then relate this result to the channel capacity. The results apply without the knowledge of the exact capacity achieving input distributions. A primary application of our results is that we can employ any lower bound derived on the capacity of the first channel (synchronization error channel in the decomposition) to find lower bounds on the capacity of the (original) noisy channel with synchronization errors. We apply the general ideas to several specific classes of channels such as synchronization error channels with erasures and substitutions, with symmetric -ary outputs and with AWGN explicitly, and obtain easy-to-compute bounds. We illustrate that, with our approach, it is possible to derive tighter capacity lower bounds compared to the currently available bounds in the literature for certain classes of channels, e.g., deletion/substitution channels and deletion/AWGN channels (for certain signal to noise ratio (SNR) ranges).

{keywords}

Synchronization errors, insertion/deletion channels, channel capacity, achievable rates.

I Introduction

Depending on the transmitting medium and the particular design, different limiting factors degrade the performance of a general communication system. For instance, imperfect alignment of the transmitter and receiver clocks may be one such factor resulting in a synchronization error channel modeled typically through insertion and/or deletion of symbols. Other factors include the effects of additive noise at the receiver etc. The main objective of this paper is to study the combined effects of the synchronization errors and additive noise type impairments and in particular to “decouple” the effects of the synchronization errors from the other parameters and obtain expressions relating the channel capacity of the combined model and the synchronization error-only channel.

We focus on achievable rates for channels which can be considered as a concatenation of two independent channels where the first one is a binary channel suffering only from synchronization errors and the second one is either a memoryless binary input symmetric -ary output channel (BSQC) or a binary input AWGN (BI-AWGN) channel. For instance, the first channel can be a binary insertion/deletion channel and the second one can be a binary symmetric channel (BSC) or a substitution/erasure channel (a ternary output channel ). Our development starts with the ternary () and quaternary () output cases, respectively, then we generalize the results to a general -ary output case. Specifically, we obtain achievable rates for the concatenated channel in terms of the capacity of the synchronization error channel by lower bounding the information rate of the concatenated channel for input distributions which achieve the capacity of the synchronization error-only channel and the parameters of the memoryless channel. The lower bounds are derived without the use of the exact capacity achieving input distribution of the synchronization error channel, hence any existing lower bound on the capacity (of the synchronization error-only channel) can be employed to obtain an achievable rate characterization for the original channel model of interest.

By channels with synchronization errors we refer to the binary memoryless channels with synchronization errors as described by Dobrushin in dobrushin [] where every transmitted bit is independently replaced with a random number of symbols (possibly empty string, i.e. a deletion event is also allowed), and the transmitter and receiver have no information about the position and/or the pattern of the insertions/deletions. Different specific models on channels with synchronization errors are considered in the literature. Insertion/deletion channels are used as common models for channels with synchronization errors, e.g., the Gallager insertion/deletion channel gallager [], the sticky channel sticky [] and the segmented insertion/deletion channel liu2010segmented [].

Dobrushin dobrushin [] proved that Shannon’s theorem applies for a memoryless channel with synchronization errors by demonstrating that information stability holds for memoryless channels with synchronization errors. That is, for the capacity of the synchronization error channel, we can write , where and are the transmitted and received sequences, respectively, and is the length of the transmitted sequence. Therefore, the information and transmission capacities of the memoryless channels with synchronization errors are equal and we can employ any lower bound on the information capacity as a lower bound on the transmission capacity of a channel with synchronization errors.

There are many papers deriving upper and/or lower bounds on the capacity of the insertion/deletion channels, e.g., see drinea2007improved [], drinea2007 [], dario [], kanoria [], asymptotic [], IT-paper []; however, only a very few results exist for insertion/deletion channels with substitution errors, e.g. gallager [], dario2 [] or in the presence of AWGN, e.g. zeng2005bounds [], junhu []. Our interest is on the latter, in fact, on more general models incorporating erasures as well as -ary channel outputs.

Let us review some of the existing relevant results on insertion/deletion channels in a bit more detail. In gallager [], Gallager considers a channel model with substitution and insertion/deletion errors (sub/ins/del) where each bit gets deleted with probability , replaced by two random bits with probability , correctly received with probability , and changed with probability , and derives a lower bound on the channel capacity (in bits/use) given by

(1)

where denotes logarithm in base 2. Fertonani and Duman in dario2 [] develop several upper and lower bounds on the capacity of the ins/del/sub channel, where they employ a genie-aided decoder that is supplied with side information about some suitably selected random processes. Therefore, an auxiliary memoryless channel is obtained in such a way that the Blahut-Arimoto algorithm (BAA) can be employed to obtain upper bounds on the capacity of the original channel. Furthermore, it is shown that by subtracting some quantity from the derived upper bounds which is, roughly speaking, more than extra information provided by the side information, lower bounds on the capacity can also be derived. In junhu [], Monte Carlo simulation based results are used to estimate information rates of different insertion and/or deletion channels in the absence or presence of intersymbol interference (ISI) in addition to AWGN with independent uniformly distributed (i.u.d.) input sequences. In zeng2005bounds [], the synchronization errors are modeled as a Markov process and simulation results are used to compute achievable information rates of an ISI channel with synchronization errors in the presence of AWGN. In IT-paper [], Rahmati and Duman compute analytical lower bounds on the capacity of the i.i.d. del/sub and i.i.d. del/AWGN channels, by lower bounding the mutual information rate between the transmitted and received sequences for i.u.d. input sequences focusing on small deletion probabilities.

The paper is organized as follows. In Section II, we formally give the models for binary input symmetric -ary output channels with synchronization errors and BI-AWGN channels with synchronization errors. In III, we give two lemmas and one proposition which will be useful in the proof of the result on BSQC channels with synchronization errors. In Section IV, we initially focus on a substitution/erasure/synchronization error channel (abbreviated as sub/ers/synch channel) which is a binary input symmetric ternary output channel, and then on a binary input symmetric quaternary output channel. After that we extend the results to the case of more general symmetric -ary output channels. In Section V, we lower bound the capacity of a synchronization error channel with AWGN (abbreviated as AWGN/synch channel) in terms of the capacity of the underlying synchronization error only channel. More precisely, we generalize the results on BSQC channels with synchronization errors when goes to infinity. We present several numerical examples illustrating the derived results in Section VI. Finally, we conclude the paper in Section VII.

Ii Channel Models

A general memoryless channel with synchronization errors dobrushin [] is defined via a stochastic matrix where is the input alphabet (e.g., for a binary input channel ), and is (possibly empty) the set of output symbols, , and . As a particular instance of this channel, if ( denoting the null string) and , we obtain an i.i.d. deletion channel.

Ii-a Binary Input Symmetric -ary Output Channel with Synchronization Errors

By a binary input symmetric -ary output channel (BSQC) with synchronization errors, we refer to a channel which can be considered as a concatenation of two independent channels, depicted in Fig 1, such that the first one is a channel with only synchronization errors with input sequence and output sequence , and the second one is a BSQC with input sequence and output sequence , where by a symmetric channel we refer to the definition given in [gallager_book, , p. 94]. In other words, a channel is symmetric if by dividing the columns of the transition matrix into sub-matrices, in each sub-matrix, each row is a permutation of any other row and each column is a permutation of any other column. For example, a channel with independent substitution, erasure and synchronization errors (sub/ers/synch channel) can be considered as a concatenation of a channel with only synchronization errors with input sequence and output sequence and a substitution/erasure channel (binary input ternary output channel) with input sequence and output sequence . In a substitution/erasure channel, each bit is independently flipped with probability or erased with probability , as illustrated in Fig. 2. (a).

Fig. 1: Binary input symmetric -ary output channel with synchronization errors.
Fig. 2: (a) Input-output relation in the substitution/erasure channel ( for all ). (b) Input-output relation in the binary input quaternary output channel ( for all ).

Another example is a binary input symmetric quaternary output channel with synchronization errors which can be decomposed into two independent channels such that the first one is a memoryless synchronization error channel and the second one is a memoryless binary input symmetric quaternary output channel shown in Fig. 2. (b).

Ii-B BI-AWGN Channels with Synchronization Errors

In a BI-AWGN channel with synchronization errors, bits are transmitted using binary phase shift keying (BPSK) and the received signal contains AWGN in addition to the synchronization errors. As illustrated in Fig. 3, this channel can be considered as a cascade of two independent channels where the first one is a synchronization error channel and the second one is a BI-AWGN channel. We use to denote the input sequence to the first channel which is a BPSK modulated version of the binary input sequence , i.e., and to denote the output sequence of the first channel and input to the second one. is the output sequence of the second channel that is the noisy version of , i.e.,

where ’s are i.i.d. zero mean Gaussian random variables with variance , and and are the received and transmitted bits of the second channel, respectively.

Fig. 3: AWGN channel with synchronization errors.

Ii-C Simple Example of a Synchronization Error Channel Decomposition into Two Independent Channels

The procedure in finding the capacity bounds used in this paper can be employed for any channel which can be decomposed into two independent channels such that the first one is a memoryless synchronization error channel and the second one is a symmetric memoryless channel with no effect on the length of the input sequence. Therefore, if we can decompose a given synchronization error channel into two channels with described properties, we can derive lower bounds on the capacity of the synchronization error channel. The advantage of this decomposition is that decomposing the original synchronization error channel into a well characterized synchronization error channel and a memoryless channel could be done in such a way that lower bounding the capacity of the new synchronization error channel be simpler than the capacity analysis of the original synchronization error channel. In Table I, we provide an example of a hypothetical channel with synchronization errors that can be decomposed into a different synchronization error channel and a memoryless binary symmetric channel (BSC). In Table II, the two channels used in the decomposition are given.

0
1
TABLE I: Transition probabilities of the hypothetical synchronization error channel.
0 0 1 0 0 1
TABLE II: Transition probabilities of two independent channels giving rise to the synchronization error channel given in Table  I.

Iii Entropy Bounds for Binary Input -ary Output Channels with Synchronization Errors

In the following two lemmas, we provide a lower bound on the output entropy and an upper bound on the conditional output entropy of the binary input -ary output channel in terms of the the corresponding output entropies of the synchronization error channel, respectively. We then give a proposition that will be useful in the proof of the result on BSQC channels with synchronization errors (note that the following two lemmas hold for any binary input -ary output channels with synchronization errors regardless of any symmetry).

Lemma 1.

In any binary input -ary output channel with synchronization errors and for all non-negative integer values of , we have

(2)

where is the random variable denoting the length of the received sequence, denotes the output sequence of the synchronization error channel and the input sequence of the binary input -ary output channel, and denotes the output sequence of the binary input -ary output channel.

Proof:

By using two different expansions of , we have

(3)

Hence, we can write

(4)

where we used the fact that by knowing , random variable is also known, i.e. . By using the same approach for , we have

(5)

Finally, we can write

(6)

where . On the other hand, due to the definition of the entropy, we can write

where denotes the expected value with respect to the random variable . Now due to the fact that is a convex function of , we apply Jensen’s inequality to write

(7)

By substituting this result into (6), the proof follows. ∎

Lemma 2.

In any binary input -ary output channel with synchronization errors and for any input distribution, we have

(8)

where denotes the -th output bit of the synchronization error channel and -th input bit of the binary input -ary output channel and denotes the output symbol of the binary input -ary output channel corresponding to the input bit .

Proof:

For the conditional output entropy, we can write

(9)

where the last equality follows since form a Markov chain. Therefore,

(10)

On the other hand, by using the fact that by knowing , is also known, we have

(11)

Furthermore, since the second channel is memoryless, we obtain

(12)

which concludes the proof. ∎

By combining the results of Lemmas 1 and 2, we obtain

(13)

which gives a lower bound on the mutual information between the transmitted and received sequences of the concatenated channel in terms of the mutual information between the transmitted and received sequences of the synchronization error channel .

Proposition 1.

For any , and forming a Markov chain , if

where is a constant, then the capacity of the channels () and () satisfy

(14)
Proof:

Using the input distribution which achieves the capacity of the channel , , we can write

(15)

Hence, for the capacity of the channel , we have

(16)

Due to the result in (13) and the result of Proposition 1, the capacity of the concatenated channel can be lower bounded in terms of the capacity of the synchronization error channel and the parameters of the second (memoryless) channel.

Iv Achievable Rates over Binary Input Symmetric -ary Output Channels with Synchronization Errors

In this section, we focus on BSQC channels with synchronization errors (as introduced in Section II-A) and provide lower bounds on their capacity. We first develop the results for sub/ers/synch channel and binary input symmetric quaternary output channel, respectively. Then give the results for general (odd and even) , respectively.

Iv-a Substitution/Erasure Channels with Synchronization Errors

The following theorem gives a lower bound on the capacity of the sub/ers/synch channel with respect to the capacity of the synchronization error channel. In a sub/ers channel, every transmitted bit is either flipped with probability of , or erased with probability of or received correctly with probability of independent of each other.

Theorem 1.

The capacity of the sub/ers/synch channel can be lower bounded by

(17)

where denotes the capacity of the synchronization error channel, , and denote the length of the transmitted and received sequences, respectively.

Before giving the proof of Theorem 1, we consider some special cases of this result. Since we have considered the general synchronization error channel model of Dobrushin dobrushin [], the lower bound (17) holds for many different models on channels with synchronization errors. A popular model for channels with synchronization errors is the Gallager’s ins/del model3 in which every transmitted bit is either deleted with probability of or replaced with two random bits with probability of or received correctly with probability of independent of each other while neither the transmitter nor the receiver have any information about the insertion and/or deletion errors. If we employ the Gallager’s model in deriving the lower bounds, for the parameter , we have

(18)

where denotes the length of the output sequence in one use of the ins/del channel, and the equality results since the channel is memoryless. By utilizing the result of (18) in (17), we obtain the following two corollaries.

Corollary 1.

The capacity of the sub/ers/ins/del channel is lower bounded by

(19)

where denotes the capacity of an insertion/deletion channel with parameters and .

Taking in this channel model gives the ins/del/sub channel, hence we have the following corollary.

Corollary 2.

The capacity of the ins/del/sub channel can be lower bounded by

(20)

To prove Theorem 1, we need the following two lemmas. In the first one we give a lower bound on the output entropy of the sub/ers/synch channel related to the output entropy of the insertion/deletion channel, while in the second one we give an upper bound on the conditional output entropy of the sub/ers/synch channel, related to the conditional output entropy of the insertion/deletion channel.

Lemma 3.

For a sub/ers/synch channel, for any input distribution, we have

(21)

where denotes the output sequence of the synchronization error channel and input sequence of the substitution/erasure channel, and denotes the output sequence of the substitution/erasure channel.

Proof:

Using the result of Lemma 1, we only need to obtain an upper bound on

for all values of . On the other hand for , we have

(22)

where denotes the number of transitions or and denotes the number of transitions or . E.g., . On the other hand, for a fixed output sequence of length with erased symbols , there are possibilities among all -tuples such that , i.e., the number of erased symbols in , and , i.e., the number of positions in and in which ’s are the flipped versions of , therefore we can write

(23)

Note that in deriving the inequality in (7), the summation is taken over the values of with . However, in (23) the summation is taken over all possible values of of length (over all -tuples), i.e. or , which results in the lower bound in (23). Furthermore, by using the fact that the probability of having erasures in a sequence of length is equal to , we obtain

(24)

By substituting this result into (2), we arrive at

(25)

concluding the proof. ∎

It is also worth noting that any capacity achieving input distribution over a discrete memoryless channel results in strictly positive output probabilities for possible output sequences of the channel ([gallager_book, , p. 95]). Therefore, for special synchronization error channel models in which for any possible length of the output sequence , all the -tuple output sequences are probable, e.g. i.i.d. deletion channel or i.i.d. random insertion channel, capacity achieving input distributions () would result in strictly positive output probability distributions for all -tuple output sequences, i.e. for all of length and all possible . Hence, the bounds in (23) and (24) can be thought as equalities for these cases.

Lemma 4.

In any sub/ers/synch channel and for any input distribution, we have

(26)
Proof:

Due to the result of Lemma 2 and the fact that in a substitution/erasure channel, regardless of the distribution of , we can write

(27)

hence the proof follows. ∎

We can now complete the proof of the main theorem.

Proof of Theorem 1 : By substituting the results of Lemmas 3 and 4 into the definition of mutual information, for the same input distribution given to both synchronization error and sub/ers/synch channels, we obtain

(28)

By using the result of Proposition 1, the proof is completed.

Iv-B Binary Input Symmetric Quaternary Output Channels with Synchronization Errors

In this subsection, we consider a binary input symmetric quaternary output channel with synchronization errors as described in Section II.

Theorem 2.

The capacity of the binary input symmetric quaternary output channel with synchronization errors can be lower bounded by

(29)

where denotes the capacity of the synchronization error only channel, and is as defined in (17).

Note that, the presented lower bound is true for all memoryless synchronization error channel models. Therefore, similar to the sub/ers/synch channel we can specialize the results to the Gallager insertion/deletion channel as given in the following corollary.

Corollary 3.

The capacity of binary input symmetric quaternary output channel with insertion/deletion errors (following Gallager’s model) is lower bounded by

(30)

To prove Theorem 2, we need the two lemmas below where the first one gives a lower bound on the output entropy of the binary input quaternary output channel with synchronization errors related to the output entropy of the synchronization error channel, and the second one gives an upper bound on the conditional output entropy of the binary input quaternary output channel with synchronization errors, related to the conditional output entropy of the synchronization error channel.

Lemma 5.

In any binary input quaternary output channel with synchronization errors and for any input distribution, we have

(31)

where denotes the output sequence of the synchronization error channel and input sequence of the binary input quaternary output channel, and denotes the output sequence of the binary input quaternary output channel corresponding to the input sequence .

Proof:

Similar to the proof of Lemma 3, we use the result of Lemma 1 by taking the summation over all possible sequences of length , i.e., regardless of or , which results into a looser lower bound. On the other hand, for , we have

(32)

where denotes the number of transitions or , denotes the number of transitions or , and denotes the number of transitions or . E.g., . Furthermore, for a fixed output sequence of length with symbols, symbols, symbols and symbols, there are possibilities among all -tuples (for ) such that , , and . By defining , i.e., the number of the times or , and , i.e., the number of the times or , we can write

(33)

By taking the summation over all possible output sequences of length , and using the fact that the probability of having the output with length containing or is , we obtain