DataRate Driven Transmission Strategy for Deep Learning Based Communication Systems
Abstract
Deep learning (DL) based autoencoder is a promising architecture to implement endtoend communication systems. In this paper, we focus on the fundamental problems of DLbased communication systems, including high rate transmission and performance analysis. To address the limited data rate issue, we first consider the error rate constraint and design a transmission algorithm to adaptively select the transmission vectors to maximize the data rate for various channel scenarios. Furthermore, a novel generalized data representation (GDR) scheme is proposed to improve the data rate of DLbased communication systems. Then, we analyze the effect of signaltonoise ratio (SNR) and mean squared error performance of the proposed DLbased communication systems. Finally, numerical results show that the proposed adaptive transmission and GDR schemes achieve higher data rate and have lower training complexity than the conventional onehot vector scheme. Both the new schemes and the conventional scheme have comparable block error rate (BLER) performance. According to both theoretical analysis and simulated results, it is suggested that low or widerange training SNR is beneficial to attain good BLER performance for practical transmission with various channel scenarios.
I Introduction
To ensure high demand for various communication applications and services, the nextgeneration network must be able to deliver enhanced mobile broadband, ultrareliable and lowlatency communications (URLLC), and massive Internet of Things (IoT) ecosystems [1, 2, 3, 4]. The primary concern is to satisfy the exponential rise in the number of user equipments and the traffic capacity of future communication systems. Hence, several promising technologies have been proposed, and they include massive multiinput and multioutput (MIMO) transmissions, millimeter wave (mmWave) communications, ultradense networks (UDNs) [5, 6, 7, 8, 9]. However, there exist a number of limitations for conventional communication systems, such as unavailable channel state information in complex transmission scenario, high complexity to process big data, and suboptimal performance caused by conventional block structure. For this reason, with the significant development of deep learning (DL) [10, 11, 12], researchers are attempting to apply the machine learning (ML), especially DL technologies, to communication system design for new benefits [13, 14, 15, 16] that cannot be obtained using the conventional approaches.
As a promising technique, deep learning applies a useful and insightful way to implement communication systems using deep neural networks (NNs). Different from the conventional communication system that consists of multiple independent blocks (e.g., source/channel coding, modulation, channel estimation, equalization), the DLbased communication system can jointly optimize transmitter and receiver for endtoend performance without block structure [17, 18]. DLbased system design is promising for future communications with the following reasons:

A DLbased communication system is optimized for endtoend performance by using deep NNs, which is fundamentally different from the blockstructure conventional communication systems with suboptimal performance.

A DLbased communication system can be optimized for a practical system over any type of channel without requiring a mathematically tractable model, including the channel models that take into account of different transmission scenarios and nonlinearities.

DL algorithms provide faster processing speed than conventional communication algorithms, since the execution of NNs can be highly parallelized on concurrent architectures and can be implemented with lowprecision data types [19].
Attracted by these advantages, there have been a number of studies on DLbased communication and signal processing using stateoftheart tools and hardware [20, 21, 22, 23, 17, 24, 25, 18, 26, 27, 28, 29, 30, 31, 32, 33, 34]. The DL method is used to deal with certain challenges in existing communication systems. For example, the DLbased belief propagation algorithm was originally used to improve the performances of channel decoding, where low complexity and near optimal decoder performance were obtained [20, 21, 22]. Around the same time, autoencoder was developed to address the problem of learning an efficient physical layer [23]. In DL theory, an autoencoder describes a deep NN in order to find a lowdimensional representation of its input at certain intermediate layer that allows reconstruction at the output with minimal error [35, Ch. 14]. The DLbased communication system can be represented and implemented by an autoencoder that is trained using the dataset off line. Then, the trained autoencoder can be directly applied to practical systems on line. A DLbased communication system interpreted as an autoencoder performs an endtoend reconstruction task that jointly optimizes transmitter and receiver as well as learns signal encoding [23, 17, 24, 32]. Considering the advantages of the autoencoder, a complete communication system represented as an autoencoder was proposed to address the challenges of frame synchronization [18, 26], where a competitive performance can be achieved even without extensive hyperparameter tuning. More recently, DLbased algorithm has been used to solve the channel state information feedback and channel estimation problems in massive MIMO systems, and it outperforms the stateoftheart compressive sensing based algorithms [27, 28, 29].
For future communication systems, there is a huge demand for data rate due to increased number of communication devices and equipment types, and improved quality of services (QoS). Consequently, high datarate schemes should be developed in DLbased communication systems for future wireless networks. However, in [17, 15, 18, 24, 25, 26, 22, 32], onehot vector, being the only data representation, has a low data rate in DLbased communication systems. The reason is that an onehot vector consists of s in all entries with the exception of a single [36], e.g., , and there are only possible transmitted messages that lead to limited data rate, which is a barrier for developing DLbased communication systems in the future. Besides, the autoencoder with onehot vector is trained with a fixed vector size , which has low flexibility for various communication systems. On the other hand, the conventional autoencoder is trained under a fixed signaltonoise ratio (SNR) with unrealistic expectation to operate well for a wide range of SNR values in practical transmission scenarios. It was reported that training the autoencoder at different SNR values will lead to various autoencoder performances [17], but there is yet a detailed study on the effect on such a system. Therefore, our objective is to design a new transmission scheme and replace the conventional onehot vector scheme in order to achieve high data rate and flexibility. As well, we will investigate the effect of training SNR on the performance of DLbased communication systems.
In this paper, an adaptive transmission algorithm is first designed for different communication scenarios to maximize the data rate in DLbased communication systems with a QoS constraint.
Then, we propose a generalized data representation (GDR) scheme to improve the data rate of DLbased communication systems.
Finally, we analyze the effect of SNR and mean squared error (MSE) performance in DLbased communication systems.
Comparable block error rate (BLER) performance can be achieved by the proposed transmission schemes with low complexity and high data rate, when compared with the conventional DLbased communication system
The major contributions of this paper are summarized as follows:

In DLbased communication systems, we formulate the data rate problem of the conventional onehot vector scheme. To address this issue, we design an adaptive transmission algorithm for different channel scenarios, where the QoS constraint is considered. In the proposed algorithm, the optimal transmission vectors are adaptively selected for different SNR conditions, where the goal is to maximize the data rate with the constraint on MSE performance. It is shown that the proposed adaptive transmission algorithm has large BLER performance gain compared with the conventional onehot vector scheme with the same data rate.

Furthermore, a generalized data representation scheme is proposed to replace the conventional onehot vector scheme. The proposed scheme can improve the data rate in DLbased communication systems. As expected, high data rate is obtained by our GDR scheme with comparable BLER performance and low complexity. To the best of the authors’ knowledge, this is the first time that the GDR scheme is proposed and its effectiveness is also verified.

We investigate the effect of SNR on the system performances in DLbased communication systems. Simulation results show that the high training SNR can improve the convergence performance in training, but it can also degrade the BLER performance in practical transmission. As a tradeoff, we introduce a widerange training SNR strategy, which shows great performance balance in convergence and BLER performance. Furthermore, it is proved that training the autoencoder at low SNR can achieve BLER and MSE performance gains when the trained autoencoder is applied to high SNR scenario. These results provide a reliable guidance to select the suitable training SNR for the optimal system performance.
The rest of this paper is organized as follows. In Section II, we describe the system model of the DLbased communication system. Section III presents an adaptive transmission algorithm. Section IV proposes the generalized data representation scheme for DLbased communication systems. Section V investigates the effect of SNR and analyzes the MSE performance of the autoencoder. In Section VI, we show the numerical results of the proposed schemes and system performances. Section VII concludes this paper.
We use the following notations throughout this paper: is a matrix; is a vector; is a scalar; is a set; is the Frobenius norm of matrix ; is the th row of matrix ; is the th element of vector a; is the zero vector; is the identity matrix; is used to denote expectation; is the floor operation; denotes the number of combinations when choosing out of .
Ii Deep LearningBased Communication Systems
In this section, we describe the DLbased autoencoder for endtoend communication systems and then provide the research motivations of this paper.
Iia Autoencoder for EndtoEnd Communication Systems
Name  Function  


Linear  
ReLU  max  
Softmax  

MSE  
Categorical crossentropy 
We consider a DLbased communication system represented as an autoencoder that consists of transmitter, channel, and receiver as shown in Fig. 1, where its NN structure is shown below. The autoencoder describes a deep NN that applies unsupervised learning in order to reconstruct the input at the output [35, Ch. 14]. At the transmitter, the message is first transformed to a vector after the vector expression processing, where . For example, if the message is transmitted, the corresponding vector expression is a onehot vector in a conventional DLbased communication system [17]. Then, the multiple dense layers, including rectified linear unit (ReLU) layer and linear layer, apply the transformation to produce the transmitted signal for discrete channel uses [18]. The commonly used activation functions are shown in TABLE I. Finally, the normalization layer ensures the power constraint of the transmitted signal as ().
The Rayleigh fading channel is implemented by a noise layer with its output being the received signal that can be given by
(1) 
where is the fading coefficient with a Rayleigh distribution, denotes the additive white Gaussian noise (AWGN) vector with a fixed variance , and where is the data rate, is the energy per bit, and denotes the noise power spectral density. Notably, there is no complex operation in the existing NN architectures, and the complex number is represented by two real numbers [17]. Here, we assume that the channel coefficients are realvalued. Furthermore, the autoencoderrepresented communication system is available for any type of channel with/without a mathematically tractable model [17, 18].
At the receiver, the received signal is passed through the ReLU layer to realize the transformation . The last layer of the receiver has a softmax activation as shown in TABLE I, which is a generalization of the logistic function that compresses an dimensional vector of arbitrary real values to an dimensional probability vector , where each element lies in the range (0, 1], and all the elements add up to 1 [35]. For the autoencoder scheme with conventional onehot vector, the estimated message is derived from the index of the element having the highest probability in . Here, the BLER of DLbased communication systems is defined as
(2) 
The autoencoder based communication system can be trained on a large training dataset off line, while the iterative training process depends on the value of loss function in each iteration. The most common loss functions are MSE and categorical crossentropy as shown in TABLE I, and these loss functions are determined by the vector expression and the probability vector . The training parameters of the autoencoder are produced to minimize the loss function. Furthermore, the trained autoencoder with the fixed parameters is applied to various practical communication scenarios on line.
IiB Motivations
Onehot vector is the conventional data representation with only one nonzero element. Thus, the data rate of the conventional DLbased communication system with onehot vector is limited to
(3) 
Over the last few years, the demand for high data rates has experienced unprecedented growth in communication systems [1, 2]. Therefore, providing a high data rate is essential for DLbased communication systems in future communications.
To improve the data rate, two different research problems can be formulated as follows:

Adaptive transmission scheme. For the conventional onehot vector scheme, the DLbased autoencoder is trained over a fixedsize transmission vector with dimension at fixed SNR scenario, which leads to two main limitations. On the one hand, the trained autoencoder for a certain value of cannot work in the scenarios with different values of . On the other hand, the performance of DLbased communication system will not be optimal when the trained autoencoder is applied to different SNR scenarios. For these reasons, there is a need for an adaptive transmission strategy for the autoencoder to improve the applicability and optimize the system performances, such as maximizing the data rate and satisfying the QoS constraint [37, 38].

Generalized data representation scheme. From the definition of the data rate , it is obvious that, for the same channel environment, the data rate is proportional to the number of bits being conveyed. However, the size of transmission vector cannot be infinite due to the high complexity associated with deep NNs. Therefore, a new data representation scheme is required to meet the high data rate demands in future communication systems.
Considering the two aspects above, datarate driven transmission strategies are required for DLbased communication systems to be employed in the future networks.
As for system performances, the autoencoder that is trained using a fixed SNR off line is expected to have a robust performance for a wide SNR region on line. In [17], it was found that an unaccommodated training SNR will result in the performance degradation of DLbased communication systems, but there is little theoretical analysis. Consequently, the effect of the training SNR needs to be investigated and a reliable criterion needs to be developed for selecting training SNR. Furthermore, it is necessary to provide more theoretical performance analysis for DLbased communication systems as the fundamental principles for future study.
Iii Adaptive Transmission Strategy
Considering the high data rate and general applicability, adaptive transmission technology can be employed in the DLbased communication system. In this section, the adaptive transmission algorithm is presented to maximize the data rate with the MSE constraint for different channel scenarios.
In Fig. 2, the adaptive transmission algorithm is considered to adaptively design the online transmission strategy for DLbased autoencoder represented communication systems.
Before the online transmission, the autoencoder has been trained with onehot vectors over a fixed training SNR () off line, while should be suitably large
(4)  
where is less than or equal to , and is a preset MSE threshold. Finally, the selected onehot vectors are used for the autoencoder on line over the current channel with .
The main steps of the adaptive transmission algorithm are summarized as follows:
Algorithm 1 Steps of the Adaptive Transmission Algorithm

Train the autoencoder with all possible onehot vectors off line.

Each onehot vector is transmitted through the trained autoencoder over the practical channel.

Calculate the practical MSE of each vector and select according to (4).

Transmit the message by using the selected vectors over the current channel on line.
Iv Generalized Data Representation Scheme
In this section, we propose a generalized data representation scheme to improve the data rate for DLbased communication systems.
Instead of the conventional onehot vector, bit vector is first considered to improve the data rate for DLbased communication systems. An order bit vector is defined as
(5) 
where indicates the number of nonzero elements. The bit vector provides possible messages for the transmission. In general, the number of possible symbols in the constellation diagram is a power of 2. For this reason, we only select out of possible symbols for communication.
Furthermore, for the autoencoder shown in Fig. 1, the vector expression at the transmitter is the practical probability distribution, and the probability vector at the receiver is the corresponding estimated probability distribution. The training goal of the autoencoder is optimizing to reconstruct through minimizing the loss function.
Thus, combining the form of bit vector in (5), we propose a generalized data representation as a probability distribution
(6) 
where the estimated message can be derived from the indices of elements with the highest probabilities in . The conventional onehot vector is a special case of the proposed GDR scheme when .
The data rate of the DLbased communication system can be improved by employing the proposed GDR as
(7) 
When , the data rate of the conventional onehot vector scheme in (3) is obtained. The data rate increases with , while value is suitably chosen and remains fixed. The performance gain of the proposed GDR scheme will significantly increase when the vector size increases.
The channel capacity of the proposed GDR scheme in the DLbased communication system is derived as
It is obvious that the capacity can be improved by using the proposed GDR in the DLbased communication system.
V Performance Analysis of the Autoencoder
In this section, we provide a theoretical analysis of MSE performance for DLbased communication systems.
Va MSE Performance Analysis
In Fig. 1, the output of the ReLU layer at receiver can be written as
(9) 
where , and denote the trainable parameters of the ReLU layer, and they are defined as
(10) 
respectively, where represents the symmetric interaction term between unit and unit in Fig. 1, and is the bias term. Thus, from (1) and (9), the th element of is given by
(11) 
where is the th row of .
Next, a probability vector is derived from the softmax function at the receiver, and its th element can be written as
(12) 
From (11)(12), in the offline training processing, different will lead to different trainable parameters and , which will have effect on in (11). As a result, , the probability of the th element is directly affected by the training SNR. Also, in the online practical transmission, the trainable parameters and are constant since the autoencoder has been trained. When the autoencoder is applied to a different practical SNR scenario, it will lead to a different estimated probability vector as well. The effect of SNR will also be analyzed through simulations.
In Appendix A, it is shown that, based on (12), the probability vector at the receiver in Fig. 1 can be approximated as
(13) 
where is a diagonal matrix that is equivalent to the effect of softmax activation layer.
At the receiver, the output of the ReLU layer consists of zero and nonzero elements as shown in (11). In this paper, we aim to analyze the effect of SNR on MSE performance. While the zero elements cannot reflect the characteristic of MSE, the nonzero output of the ReLU layer is considered and can be derived from (11) as
(14) 
if
(15) 
Thus, the probability vector under the assumption of (15) can be expressed as
(16) 
where is the equivalent matrix of softmax activation layer in the nonzero case as (15).
Here, the average MSE of the DLbased communication system in the case of (15) can be given from (14) and (16) as
(17)  
where is the variance of the Rayleigh fading coefficient , and step is due to . After the autoencoder is trained over , the transformation matrices of the autoencoder are constant, where is the noise variance at the training scenario. When the trained autoencoder is applied to the practical communication scenario with , the noise variance of the current practical channel scenario is . For the nonzero case, it can be observed from (17) that, when , the practical MSE performance will be better than that of the training scenario; when , the converse is true. It indicates that the trained autoencoder can attain better system performance when it is applied to higher SNR scenario. For the zero case in (11), the variance of noise has no effect on the MSE performance. The MSE performance of the DLbased communication system will also be verified through simulations.
VB WideRange Training SNR Strategy
In conventional DLbased communication systems, the autoencoder is trained over a fixed SNR off line, which leads to limited generalization performance while facing with various communication scenarios. Here, we propose a widerange training SNR strategy by employing multiple training SNRs, and it will improve the diversity of training dataset aiming to obtain a great generalization performance. For example, the widerange training SNR can be set to dB for offline training. Also, the system performance gain of the proposed widerange training SNR strategy will be shown by simulation results.
Vi Numerical Results
Parameter  Value 

Optimizer  Adam [39] 
Loss function  MSE 
Epoch  150 
Batch size  45 
Trained samples  2 
Test samples  1 
In this section, we evaluate the numerical results of the proposed adaptive transmission algorithm, the GDR scheme, and the system performances in the DLbased communication system via simulations on the TensorFlow framework. In all simulations, the autoencoder is trained over the stochastic AWGN channel model with channel uses without exhaustive hyperparameter tuning. Here, we use the same set of parameters for the autoencoder setup as described in TABLE II. Notably, it has been proved that the DLbased communication system with onehot vector can obtain competitive system performance compared to the conventional communication system in [25, 18]; therefore, we will not compare with the conventional communication system.





Total  



TABLE III presents the simulated and theoretical number of training parameters in autoencoder, where different size of the data representation is employed. From TABLE III, it is clear that the simulated number of trainable parameters increases with from to , not only for the total number but also for the number of each layer except for the normalization layer. The simulated results agree with the theoretical number of parameters as shown in the last row of TABLE III. The increasing training parameters lead to an increasing complexity for training. For the conventional onehot vector, the data rate can be improved by increasing as shown in (3) at the cost of high complexity. While, the data rate of the proposed GDR scheme can be improved by controlling the number of nonzero elements with a small as shown in (7), which leads to a low complexity.
Via Performance of the Proposed Adaptive Transmission Scheme
In this subsection, we show the simulated BLER and MSE performance of the proposed adaptive transmission scheme in the DLbased communication system. Here, the autoencoder is trained when the training SNR is dB.
Figure 3 depicts the simulated BLER performance of the DLbased autoencoder that employs the proposed adaptive transmission scheme and the conventional onehot vector scheme, where the MSE thresholds are , , and . First, it can be seen from Fig. 3 that, the BLER of the conventional onehot vector scheme increases when is varied from to , since the smaller requests less trainable parameters as shown in TABLE III. With the same training dataset, the less trainable parameters contribute to better training accuracy. Second, for the proposed adaptive transmission scheme, the BLER increases when the MSE threshold is increased from to in Fig. 3. The reason is that, to maximize the data rate, the lower MSE threshold (means the tighter bound) requires smaller to satisfy the MSE constraint, which results in lower BLER. Then, for each MSE threshold, the number of selected vectors adaptively increases from to with the increasing practical SNR at the axis. For example, while , the adaptively changes with SNR as . For this reason, higher practical SNR makes it easy to obtain the MSE requirement and leads to larger to maximize the data rate. At last, when the data rate is the same in Fig. 3, the BLER performance of the adaptive transmission scheme is better than that of the onehot vector scheme. The reason for the performance gain is that the proposed adaptive transmission algorithm can select the optimal vectors that meet the MSE requirement as shown in (4).
Figure 4 illustrates the data rate performance of the autoencoder that employs the conventional onehot vector scheme and the proposed adaptive transmission algorithm with the MSE thresholds being , and . From Fig. 4, we see that the data rates of the conventional onehot vector scheme are constant for all practical values of SNR, which cannot obtain the maximum data rate with the great BLER performance at the same time. However, in Fig. 4, the data rate of the proposed adaptive transmission scheme increases with SNR due to the increasing as shown in Fig. 3, which is consistent with (4). By comparing Fig. 3 with Fig. 4, the proposed adaptive transmission algorithm can obtain the better BLER performance than that of the conventional onehot vector scheme when operating at the same data rate.
Figure 5 presents the simulated MSE performance for the practical communication system that employs the proposed adaptive transmission algorithm with the MSE thresholds being , and . It is seen from Fig. 5 that the simulated MSE of the proposed adaptive transmission scheme increases with the MSE threshold. Furthermore, the simulated MSE of the proposed scheme decreases while the practical SNR increases, which is consistent with the result in (17). As expected, when the simulated MSE reaches the corresponding MSE threshold, the number of the selected vectors is almost which is the maximum value, and the maximum data rate is obtained.
ViB Performance of the Proposed GDR Scheme
This subsection shows the BLER performance and the channel capacity of the proposed GDR scheme in the DLbased communication system, where the training SNR is dB.
Figure 6 shows the simulated BLER performance of the DLbased communication system that employs the proposed GDR and conventional onehot vector schemes for comparison. There are a number of observations can be seen from Fig. 6. First, with the same data rate (bits/channel use) including the proposed schemes with , with , and the conventional scheme with , the proposed GDR schemes have better BLER performance with lower complexity than that of the conventional onehot vector scheme. Obviously, the BLER decreases with the vector size for the same reason as that in Fig. 3. Second, with the same vector size , the proposed GDR schemes ( and ) obtain comparable BLER performances when compared to the conventional onehot vector scheme (). It indicates that, with the same vector size, the number of nonzero elements has little effect on the BLER performance. With the similar BLER, the data rates of the GDR schemes are (bits/channel use) while and (bits/channel use) while . In both cases, the data rate is greater than that of the conventional onehot vector scheme as (bits/channel use). Third, the simulated BLER is less than when the SNR is greater than 2 dB in Fig. 6, which demonstrates that the autoencoder obtains a high accuracy by sufficient training over dB.
Figure 7 illustrates the channel capacity of the DLbased communication system employing different data representations. It can be seen from Fig. 7 that, with , the channel capacity increases when the order increases from to , which is consistent with the result in (IV). This shows that the proposed GDR scheme can obtain a remarkable channel capacity improvement. Notably, the performance gain of the proposed GDR scheme is increased when the vector size increases. As shown in Fig. 7, the GDR scheme employing with has a great performance gain when compared with the conventional scheme employing with . Besides, the channel capacity of proposed GDR schemes ( with and with ) is the same as that of the conventional onehot vector scheme (, ) in Fig. 7. To obtain the same capacity with GDR scheme, the conventional onehot vector scheme needs to improve the vector size , which leads to a BLER performance degradation as shown in Fig. 6.
ViC Performance Comparison of Different Training SNR
In this subsection, we investigate the effect of training SNR on system performance for the DLbased communication system, including the loss function performance in training process, and the simulated BLER as well as MSE performances in practical transmission process. Here, the data representation parameters are and .
Figure 8 shows the simulated loss function performance in training processing, while the autoencoder is trained over different SNRs and widerange SNR. The widerange SNR is set as dB, which includes all the fixed SNRs except for dB. Here, an epoch is the process that training dataset totally passes through the autoencoder once. In Fig. 8, with the fixed training SNR increasing from dB to dB, the loss value decreases and the convergence of loss function gets better, which indicates that the good channel environment contributes to improve the training performance. However, with dB, the loss value does not converge within epoches. Furthermore, it can be seen from Fig. 8 that, the loss value of the autoencoder training with widerange SNR is similar to that of the autoencoder training with dB. The simulated results suggest that the training SNR has significant effect on the training performance of the autoencoder.
Figure 9 depicts the simulated BLER performance of the practical DLbased communication system employing the trained autoencoder with different fixed training SNRs and widerange training SNR. In Fig. 9, the BLER decreases with the fixed training SNR ranging form dB to dB. The reason is that, with the lower training SNR (that is to say the worse channel environment), the autoencoder needs to learn more features to reconstruct the input at the output, which leads to a robust autoencoder and better BLER performance. However, the training SNR has a lower bound for the autoencoder. As shown in Fig. 9, while the training SNR is dB, the BLER is approximately 0.6, which demonstrates that the autoencoder trained over this channel environment cannot learn the features anymore. It is consistent with the nonconvergence performance of the loss function with dB in Fig. 8. Besides, Fig. 9 shows that the BLER performance of the widerange training SNR scheme is similar to that of dB scheme, which is almost the best performance except for the dB scheme. It shows that training with widerange SNR can improve the generalization performance of the autoencoder. Combining Fig. 8 with Fig. 9, it can be found that, with the higher training SNR, the autoencoder obtains better convergence performance in training but worse BLER performance in testing. The simulated results indicate that the training SNR will directly affect the system performance, which agrees with the analysis in Subsection VA.
Figure 10 illustrates the simulated MSE performance of the practical DLbased communication system employing different trained autoencoders, while the training SNRs include different fixed SNRs and widerange SNR. In Fig. 10, it is clear that the MSE decreases when the practical SNR at the axis increases. It indicates that the MSE performance improves when the trained autoencoder is applied to a higher SNR scenario, which is consistent with the analysis in (17). Furthermore, the simulated MSE performance in Fig. 10 is similar to the BLER performance as shown in Fig. 9 for the same reasons.
Vii Conclusion
In this letter, we proposed new transmission schemes to address the problem of limited data rate and investigated the system performances of the DLbased communication system. We designed an adaptive transmission scheme for different channel scenarios to maximize the data rate with the error rate constraint. Furthermore, the GDR scheme was proposed, and it can obtain higher data rate and comparable BLER performance with a lower complexity when compared with the conventional onehot vector scheme. Besides, the effect of training SNR and MSE performance were analyzed and verified by simulations. We discovered that high training SNR can lead to good convergence in training process but worse BLER performance for practical transmission. We also introduced a widerange training SNR strategy to address the tradeoff between convergence and error rate. On the other hand, it is shown that the autoencoder trained over a low SNR can attain better BLER and MSE performances when operating in the high SNR region. As a result, it is concluded that training the autoencoder at low SNR will lead to good system performance.
Appendix A Derivation of (13)
Next, can be approximated according to the Taylor’s theorem as
(21) 
where is a sufficiently large integer.
Footnotes
 Notably, throughout this paper, the conventional DLbased communication system refers to the autoencoder based communication system that employs the onehot vector.
 If is too large, the training complexity is prohibitive since the autoencoder must see every message once at least [17].
References
 F. Boccardi, R. W. Heath, A. Lozano, T. L. Marzetta, and P. Popovski, “Five disruptive technology directions for 5G,” IEEE Commun. Mag., vol. 52, no. 2, pp. 74–80, Feb. 2014.
 M. Agiwal, A. Roy, and N. Saxena, “Next generation 5G wireless networks: A comprehensive survey,” IEEE Commun. Surveys Tut., vol. 18, no. 3, pp. 1617–1655, 3rd Quart. 2016.
 P. Popovski, “Ultrareliable communication in 5G wireless systems,” in Proc. 1st Int. Conf. 5G Ubiq. Connect., Nov. 2014, pp. 1–6.
 Z. Dawy, W. Saad, A. Ghosh, J. G. Andrews, and E. Yaacoub, “Toward massive machine type cellular communications,” IEEE Wireless Commun., vol. 24, no. 1, pp. 120–128, Feb. 2017.
 J. Hoydis, S. ten Brink, and M. Debbah, “Massive MIMO in the UL/DL of cellular networks: How many antennas do we need?” IEEE J. Sel. Areas Commun., vol. 31, no. 2, pp. 160–171, Feb. 2013.
 E. G. Larsson, O. Edfors, F. Tufvesson, and T. L. Marzetta, “Massive MIMO for next generation wireless systems,” IEEE Commun. Mag., vol. 52, no. 2, pp. 186–195, Feb. 2014.
 Z. Pi and F. Khan, “An introduction to millimeterwave mobile broadband systems,” IEEE Commun. Mag., vol. 49, no. 6, pp. 101–107, June 2011.
 X. Ge, S. Tu, G. Mao, C. Wang, and T. Han, “5G ultradense cellular networks,” IEEE Wireless Commun., vol. 23, no. 1, pp. 72–79, Feb. 2016.
 H. Zhang, Y. Dong, J. Cheng, M. J. Hossain, and V. C. M. Leung, “Fronthauling for 5G LTEU ultra dense cloud small cell networks,” IEEE Wireless Commun., vol. 23, no. 6, pp. 48–53, Dec. 2016.
 X. Chen and X. Lin, “Big data deep learning: Challenges and perspectives,” IEEE Access, vol. 2, pp. 514–525, May 2014.
 J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015.
 Q. Mao, F. Hu, and Q. Hao, “Deep learning for intelligent wireless networks: A comprehensive survey,” IEEE Commun. Surveys Tut., vol. 20, no. 4, pp. 2595–2621, 4th Quart. 2018.
 D. Yu and L. Deng, “Deep learning and its applications to signal and information processing, exploratory DSP,” IEEE Signal Process. Mag., vol. 28, no. 1, pp. 145–154, Jan. 2011.
 C. Jiang, H. Zhang, Y. Ren, Z. Han, K. Chen, and L. Hanzo, “Machine learning paradigms for nextgeneration wireless networks,” IEEE Wireless Commun., vol. 24, no. 2, pp. 98–105, Apr. 2017.
 T. Wang, C. K. Wen, H. Wang, F. Gao, T. Jiang, and S. Jin, “Deep learning for wireless physical layer: Opportunities and challenges,” China Commun., vol. 14, no. 11, pp. 92–111, Nov. 2017.
 X. You, C. Zhang, X. Tan, S. Jin, and H. Wu, “AI for 5G: Research directions and paradigms,” Science China Information Sciences, Sep. 2018.
 T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical layer,” IEEE Trans. Cogn. Commun. Netw., vol. 3, no. 4, pp. 563–575, Dec. 2017.
 S. Dörner, S. Cammerer, J. Hoydis, and S. t. Brink, “Deep learning based communication over the air,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 132–143, Feb. 2018.
 V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of neural networks on CPUs,” in Deep Learn. Unsupervised Feature Learn. NIPS Workshop, 2011.
 E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in IEEE Annu. Allerton Conf. Commun. Control Comput. (Allerton), Sep. 2016, pp. 341–346.
 E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, and Y. Beï¿½ï¿½ery, “Deep learning methods for improved decoding of linear codes,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 119–131, Feb. 2018.
 T. Gruber, S. Cammerer, J. Hoydis, and S. t. Brink, “On deep learningbased channel decoding,” in IEEE Annu. Conf. Inf. Sci. Syst. (CISS), Mar. 2017, pp. 1–6.
 T. O’Shea, K. Karra, and T. Clancy, “Learning to communicate: Channel autoencoders, domain specific regularizers, and attention,” in IEEE Int. Symp. Signal Process. Inf. Technol., Dec. 2016, pp. 1–6.
 T. Erpek, T. J. O’Shea, and T. C. Clancy, “Learning a physical layer scheme for the MIMO interference channel,” in IEEE Int. Conf. Commun (ICC), May 2018, pp. 1–5.
 T. J. Oï¿½ï¿½Shea, T. Roy, and T. C. Clancy, “Overtheair deep learning based radio signal classification,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 168–179, Feb. 2018.
 A. Felix, S. Cammerer, S. Dörner, J. Hoydis, and S. t. Brink, “OFDMautoencoder for endtoend learning of communications systems,” arXiv preprint arXiv:1803.05815, 2018.
 C. K. Wen, W. T. Shih, and S. Jin, “Deep learning for massive MIMO CSI feedback,” IEEE Wireless Commun. Lett., to be published, DOI 10.1109/LWC.2018.2832128.
 H. He, C. Wen, S. Jin, and G. Y. Li, “Deep learningbased channel estimation for beamspace mmwave massive MIMO systems,” IEEE Wireless Commun. Lett., vol. 7, no. 5, pp. 852–855, Oct. 2018.
 T. Wang, C. Wen, S. Jin, and G. Y. Li, “Deep learningbased CSI feedback approach for timevarying massive MIMO channels,” IEEE Wireless Commun. Lett., to be published, DOI 10.1109/LWC.2018.2874264.
 N. Samuel, T. Diskin, and A. Wiesel, “Deep MIMO detection,” arXiv preprint arXiv: 1706.01151, 2017.
 M. Kim, N. Kim, W. Lee, and D. Cho, “Deep learningaided SCMA,” IEEE Commun. Lett., vol. 22, no. 4, pp. 720–723, Apr. 2018.
 S. Xue, Y. Ma, N. Yi, and R. Tafazolli, “Unsupervised deep learning for MUSIMO joint transmitter and noncoherent receiver design,” IEEE Wireless Commun. Lett., to be published, DOI 10.1109/LWC.2018.2865563.
 K. Kim, J. Lee, and J. Choi, “Deep learning based pilot allocation scheme (DLPAS) for 5G massive MIMO system,” IEEE Commun. Lett., vol. 22, no. 4, pp. 828–831, Apr. 2018.
 H. Ye, G. Y. Li, and B. Juang, “Power of deep learning for channel estimation and signal detection in OFDM systems,” IEEE Wireless Commun. Lett., vol. 7, no. 1, pp. 114–117, Feb. 2018.
 I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016.
 A. D. Stefano, O. Mirabella, G. D. Cataldo, and G. Palumbo, “On the use of neural networks for Hamming coding,” in IEEE Int. Symp. Circuits and Systems, vol. 3, Jun. 1991, pp. 1601–1604.
 L. Wu, Z. Zhang, and H. Liu, “Adaptive modulation with finite rate feedback for QR decompositionsuccessive interference cancellationbased multiplein multipleout systems,” IET Commun., vol. 7, no. 5, pp. 456–462, Mar. 2013.
 X. Chen, L. Wu, Z. Zhang, J. Dang, and J. Wang, “Adaptive modulation and filter configuration in universal filtered multicarrier systems,” IEEE Trans. Wireless Commun., vol. 17, no. 3, pp. 1869–1881, Mar. 2018.
 D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv: 1412.6980, 2014.