An Augmented Nonlinear Complex LMS for Digital SelfInterference Cancellation in FullDuplex DirectConversion Transceivers
Abstract
In future fullduplex communications, the cancellation of selfinterference (SI) arising from hardware nonidealities, will play an important role in the design of mobilescale devices. To this end, we introduce an optimal digital SI cancellation solution for sharedantennabased directconversion transceivers (DCTs). To establish that the underlying widely linear signal model is not adequate for strong transmit signals, the impacts of various circuit imperfections, including power amplifier (PA) distortion, frequencydependent I/Q imbalances, quantization noise and thermal noise, on the performance of the conventional augmented complex least mean square (ACLMS) based SI canceller, are analyzed. In order to achieve a sufficient signaltonoiseplusinterference ratio (SNIR) when the nonlinear SI components are not negligible, we propose an augmented nonlinear CLMS (ANCLMS) based SI canceller for a joint cancellation of both the linear and nonlinear SI components by virtue of a widelynonlinear model fit. A rigorous mean and mean square performance evaluation is conducted to justify the performance advantages of the proposed scheme over the conventional ACLMS solution. Simulations on orthogonal frequency division multiplexing (OFDM)based wireless local area network (WLAN) standard compliant waveforms support the analysis.
Fullduplex, I/Q imbalance, selfinterference, augmented complex LMS (ACLMS), augmented nonlinear complex LMS (ANCLMS), mean and mean square analysis
1 Introduction
\IEEEPARstartThe fullduplex (FD) technology aims at doubling the radio link data rate through simultaneous and bidirectional communication at the same center frequency, and is widely considered as a drivingforce for more spectrally efficient wireless networks and a potential candidate to fulfill 5G’s ambition of a 1000fold gain in capacity [1, 2]. One of the major challenges in FD communications is the socalled selfinterference (SI) problem, a strong transmit signal coupled into the receiver (Rx) path. Since the transmitter (Tx) and Rx chains are closely linked together in each transceiver node of FD communication systems, the SI power leaked into and reflected from the Tx chain could be even 50 dB to 110 dB higher than the Rx sensitivity level in either wireless local area network (WLAN) or cellular scenarios [3, 4, 5]. The design of FD transceivers has long been considered impossible for practical realizations and implementations, and it is only recently that its feasibility was experimentally demonstrated using the wireless openaccess research platform (WARP) with WiFi waveforms [6, 7, 8, 9, 10, 11]. Based on this feasibility result, it was recently suggested that a preferable FD network should consist of backhaul nodes operating in the FD mode and access nodes remaining in the legacy halfduplex (HD) mode [12, 13, 14]. However, recent researcher has showed that operating access nodes in FD mode significantly leverages the gain in degrees of freedom (DoF) in either ergodic or fastfading channel [15], and an imperative is to design a hardware structure suitable for massproduction. Owing to the smallsize, lowcost and lowenergyconsumption, directconversion transceivers (DCTs) are widely applied in HD wireless systems, and are also suitable for farend device implementation in the context of FD communication systems.
In order to provide SI cancellation, there exist numerous types of hardware solutions. According to the antenna placement strategies, these can be classified into separateantennasbased and sharedantennabased schemes. When each transceiver node is equipped with more than two separate antennas, SI attenuation can be achieved by improving the electromagnetic insulation between the antennas. Owing to the inherent closedloop of FD systems, the knowledge of the SI channel matrix can be obtained by involving either placing extra transmit antennas or allocating specific spatial resources [16, 17]. On the other hand, the sharedantennabased design aims to separate the transmit and receive signals by sharing a common antenna [9, 10], the key component of which is a three port routing device, known as a circulator, used to isolate the incoming and outgoing signals. Requiring only offtheshelf RF components, the sharedantenna structure stands out as a costeffective and energysaving choice for the design of mobilescale FD transceivers, and demonstrations on the WARP have shown that even 110 dB and 103 dB SI cancellation can be respectively achieved in SISO [9] and MIMO systems[10].
In a sharedantenna structure, there is still need for further nontrivial analog and digital SI cancellation, due to the leakage of the circulator, singlepath reflection from the antenna, and multipath interference from the surrounding environment. The purpose of analog SI cancellation is to prevent the saturation of the SI power level of the Rx lownoise amplifier (LNA), and of the same time, to ensure that the difference between the power of residual SI and the received signal of interest does not exceed the dynamic range of an analogtodigital converter (ADC) [18]. Subsequently, a further digital baseband cancellation is performed to deal with the residual SI components, as well as with RF circuit nonidealities, mainly including nonlinear distortion, I/Q imbalance and phase noise. The nonlinearity is largely caused by the power amplifier (PA), while IQ imbalance and phase noise are mainly induced by the imperfect local oscillator (LO). The impact of PA nonlinear distortion on FD DCTs has been reported and investigated in [19, 20], while the effect of phase noise was analyzed in [21, 22]. Since the I/Q imbalance is essentially reflected in the mismatch between inphase and quadrature components of the complexvalued I/Q signal, it is reflected in an image interference associated with the original signal [23, 24, 25].
The impact of the image interference caused by Tx I/Q imbalance on the SI cancellation has been studied in [26], indicating that it heavily limits the receiver path signaltonoiseplusinterference ratio (SNIR). However, due to size constraints of FD DCTs, the Rx and Tx may share a common imperfect LO, therefore, a more accurate analysis of the impact of image interference on SI cancellation should be performed by a joint consideration of both Tx and Rx I/Q imbalances. Motivated by this finding, a widely linear framework was then developed, whereby not only the original transmit signal, but also its complex conjugate, i.e., the image interference, are jointly processed to form an estimate of the SI signal, followed by blockbased parameter estimation methods to estimate the cancellation parameters through widely linear leastsquares model fitting [27]. By exploiting the advantages of widely linear adaptive estimation algorithms as compared with blockbased ones, such as their lower computational complexity and faster adaptation for potential timevarying channels, the augmented (widely linear) complex least mean square (ACLMS) adaptive filtering algorithm [28, 29, 30] has been employed in a DSPassisted analog SI cancellation process, and its theoretical performance in the presence of Tx and Rx IQ imbalances has been evaluated [31]. However, for simplicity, in [31], the I/Q imbalances within transmitters and receivers were considered to be frequencyindependent, which is not the case in wideband scenarios, where their frequency selectivity has been extensively reported and justified in [24]. Furthermore, although it has been illustrated by simulations that due to the undermodeling problem, ACLMS yields suboptimal SI cancellation results in the presence of PA nonlinearity, a theoretical understanding of this suboptimality and ways of its mitigation are still lacking.
Therefore, in this paper, we first provide a comprehensive mean and mean square performance analysis of the ACLMS based SI canceller, in both transient and steadystate stages, to theoretically establish its suboptimality in wideband FD DCTs. For rigor, both the PA nonlinear distortion and frequencydependent image interferences are also considered. Next, in order to achieve a sufficient amount of SNIR when the nonlinear SI components are not negligible, an augmented nonlinear CLMS (ANCLMS) is proposed for a joint cancellation of both the image and the nonlinear SI components by virtue of a widelynonlinear model fitting, and a theoretical performance evaluation is conducted to demonstrate its performance advantages over ACLMS. Moreover, to facilitate its use in practical applications, a data prewhitening scheme is employed to speed up its convergence. Simulations on the proposed canceller with orthogonal frequency division multiplexing (OFDM)based WLAN standard compliant waveforms applied support the analysis.
Notations: Lowercase letters are used to denote scalars, a, boldface letters for column vectors, a and boldface uppercase letters for matrices, A. The superscripts , , and denote respectively the complex conjugation, transpose, Hermitian transpose and matrix inversion operation. The operator represents the trace of a matrix, while operators , , respectively denote the convolution, Kronecker product and Frobenius norm. The statistical expectation operator is denoted by , matrix determinant by , while operators and extract respectively the real and imaginary parts of a complex variable and . The operator stands for a column vector composed by the diagonal elements of the matrix A. Matrix vectorization is designated by , which returns a column vector transformed by stacking the successive columns of matrix, and its inverse operation, i.e., restoring the matrix from the its vectorized form, is denoted by . The extraction of matrix diagonal element into a vector is denoted by .
2 FullDuplex Transceiver and Its WidelyLinear Baseband Equivalent Model
2.1 Fd Dct
The structure of a typical sharedantenna FD DCT is given in Fig. 1, and due to its simplicity this structure is widely adopted in modern wireless transceivers [32]. We here briefly discuss the physics for I/Q imaging and PA distortion, the two vital impairments within a FD DCT, and we refer to [20] for a complete characterization of the effective SI waveform in different stages of the transceiver.
The I/Q imbalance impairment, characterized by an amplitude difference between the I/Q oscillators, and/or a phase shift from the nominal , is a consequence of the imperfections of I/Q mixers at both the transmitter and receiver chains. In wideband systems, the mismatch between the lowpass filters of the inphase and quadrature branches also contributes to the frequency selective nature of I/Q imbalance [33, 24]. The Tx frequencydependent I/Q imbalance process occurs in the red shaded region in Fig. 1, for which inputoutput relation in discretetime baseband can be expressed as
(1) 
where is the original SI waveform before digitalto analogconversion, perfectly known by the receiver, and is the I/Q imbalanced output of the Tx I/Q mixer. The channel impulse responses (IRs) for the direct signal and its image component , that is, and , can be further described as [24]
(2) 
The physical meaning of , ,
and are provided in Table 1. The type of transformations in (1) and (2) , where both the direct and complexconjugated
signals are filtered and finally summed together, are called
widely linear [34, 35].
The quality of the I/Q mixer can be quantified by the image rejection
ratio (IRR), defined by
Symbol  Denotation 

Tx frequencydependent IR of inphase branch  
Tx frequencydependent IR of quadrature branch  
Tx frequencyindependent gain imbalance  
Tx frequencyindependent phase imbalance  
I/Q imbalance IR of Tx direct component  
I/Q imbalance IR of Tx image component  
I/Q imbalance IR of Rx direct component  
I/Q imbalance IR of Rx image component  
Image rejection ratio  
PA gain  
Gain of nonlinear PA  
VGA Gain  
LNA Gain  
Tx IQ mixer gain  
Rx IQ mixer gain  
LNA thermal noise  
ADC quantization noise  
Memory polynomials of PA  
Estimation error of analog cancellation  
Receiver sensitivity  
SNR requirement  
Dynamic range of ADC 
Another considerable impairment within a FD DCT is the PA nonlinearity, which may occur in the green shaded region in Fig. 1. The lowcost PA, which is used in the DCT to enhance the power of a signal before transmission, inevitably causes nonlinear distortion, and the common approach to describe its wideband behavior is via the WienerHammerstein model, given by [36]
(3) 
The physical meanings for , and in (3) are provided in Table. 1. Note that the gain from Tx variable gain amplifier (VGA) is omitted, since it is linear and can be absorbed into the SI waveform . The nonlinear term on the right hand side (RHS) of (3) is the thirdorder intermodulation term, which is much stronger compared with other terms beyond thirdorder. For simplicity, we shall now omit the descriptions of other modules within the FD DCT in Fig. 1, as well as other less significant impairments, such as phase noise and timing error, to yield a simplified widely linear relation from the SI waveform in the transmitter to the digitalized signal in the receiver, given by [20]
(4) 
where and are the endtoend channel IRs for the direct component and its image counterpart , given by
(5)  
(6)  
The physical meanings of , , and are listed in Table 1. Suppose that Tx and Rx mixers have the same level of IRR at dB, then, from (5) and (6), we observe that the second term on the RHS of (5) is negligible, since it is dB weaker than the first one, and the power of is considered to be dB lower than that of . The third term on the RHS of (4), that is, , is the received signal of interest, whose power before the digital SI cancellation can be calculated as
(7) 
where is the Rx mixer gain, defined as . The explanations for the other parameters can be also found in Table. 1. The composite noise term, that is, in (4), represents the sum of interference components, including PA nonlinearity, thermal noise and quantization noise from an ADC, given by [20]
(8)  
where
(9) 
(10) 
(11) 
(12) 
(13) 
in which, represents the thirdorder intermodulated (IMD) component in (3), and is the Tx mixer gain. The approximation in (9) is made by considering a realistic situation for typical FD I/Q imbalance parameters, whereby the energy of the signal output from the Tx mixer is mainly concentrated in the lightofsight path [33]. The corresponding channel IRs for and its image counterpart are denoted respectively by and . The interference, resulted from thermal noise during the digital SI cancellation process, is represented by in (12), where denotes the thermal noise induced by a LNA, and in (13) is an image counterpart of . The quantization noise is denoted by , whose power is subject to the number of ADC bits and the peaktoaveragepowerratio (PAPR) of .
2.2 Widelylinear Modelling in a Vectorized Form and Signal Assumptions
For mathematical simplicity, we consider a vectorized form to represent the widely linear relation between the observed signal and its corresponding SI waveform in (4), given by
(14) 
where and are the optimal endtoend channel IRs, determined by transmit and receive frequencydependent I/Q imbalances, PA memory and residual of analog cancellation, as given in (5) and (6). is of length , whereby the SI waveform is a zeromean proper white Gaussian random variable with variance . The Gaussianity and properness assumptions on are valid for wideband OFDM waveforms. Indeed, the work in [37] verified that a bandlimited uncoded OFDM symbol converges to a proper Gaussian random process as the number of subcarriers increases, and its whiteness is guaranteed due to the fact that is independent temporally. The analysis in [27] illustrates that under practical conditions, the power of the image thermal noise signal is much lower than those of the other components. Therefore, based on (8), the expression for the overall noise signal can be simplified as
(15) 
in which and with respectively represent the endtoend channel IRs of the IMD SI components, that is, , and its complex conjugate . Then, from (9), we know that is a zeromean complexvalued random variable, whose variance is given by
(16) 
By assuming that is a zeromean complexvalued additive white Gaussian noise (AWGN), its variance can be determined as
(17) 
where is the sensitivity level of the receiver and is the SNR requirement, and hence, from (12), the AWGN nature of can be also guaranteed, and its variance can be obtained as
(18) 
The quantization noise is assumed to be another zeromean AWGN process with variance , and is independent of . In this way, the VGA gain , used to ensure the received signal fit within the voltage range of the ADC, can be calculated as
where is the dynamic range of the ADC.
2.3 Component Analysis
From (4), observe that in a possible FD DCT, the baseband signal before digital SI cancellation is composed of various interference components, including SI , IMD SI , thermal noise and their image counterparts, as well as quantization noise . In order to visualize which terms are counted as primary interferences, simulations were carried out to illustrate their relative powers within FD DCTs. Typical system parameters, as suggested by [27], were chosen, and they are listed in Table 2, which shows that two types of FD DCTs with practical levels of analog SI cancellation are considered. The Type 2 DCT exhibits an inferior analog SI cancellation capability than its Type 1 counterpart, resulting in a weaker received signal of interest and thermal noise. The power of the thermal noise was chosen to be 15 dB below the receiver sensitivity, while that of the received signal of interest at the input of the receiver chain was exactly at the sensitivity level, as defined in (7). The impact of the analog cancellation on the DCT is indicated by the power of the analog cancellation error vector , which is incorporated into the channel IRs , , and . The coefficients of the filters are given to ensure the mixer gain or is 6 dB and IRR level is 25 dB, a specific requirement of 3GPP LTE [38]. The PA inputreferred thirdorder intercept point (IIP3), represented by , is a fictitious point used to determine the power level of the thirdorder nonlinear distortion in the Tx output.
Parameter  Value  Notation  
Receiver Sensitivity  89 dBm  
SNR requirement  15 dB  
Thermal noise floor  104 dBm  
RF separation 


RF attenuation 


IRR  25 dB  
Tx mixer gain  6 dB  
Rx mixer gain  6 dB  
PA gain  27 dB  
PA IIP3  20 dBm  
LNA gain  25 dB  
Transmit power  dB  
ADC dynamic range  7 dB  
Quantization noise power  60 dBm 
As shown in Fig. a, in a Type 1 FD DCT, the SI component and its image counterpart were both the dominant interferences to the signal of interest in the entire transmit power range, and when the transmit power becomes higher, the IMD SI component linearly increases to be another major interference. On the other hand, as indicated by Fig. b, in a Type 2 FD DCT, when the transmit power goes above 20 dBm, the thermal noise would become weaker than the quantization noise and the image IMD SI . This is because either a stronger nonlinear SI or a less efficient analog cancellation results in a lower VGA gain, which in turn yields a weaker amplification of signal of interest and thermal noise , as indicated by (7), (17) and (LABEL:kBB).
3 Conventional ACLMS based SI Canceller and Its Performance Analysis
As discussed in Section 2, the effect of of Tx and Rx IQ imbalances is to make both the SI component and its image counterpart the dominant interferences in a FD DCT. To mitigate this issue, a widely linear model based augmented complex least mean square (ACLMS) adaptive filtering algorithm was employed in [31] as a further DSPassisted canceller after the analog SI cancellation procedure, and a preliminary SI cancellation performance analysis of ACLMS was conducted. For simplicity, the I/Q imbalances within transmitter and receiver were considered to be frequencyindependent in [31], which, however, is not the case in wideband scenarios, where frequency selectivity has been rigourously justified [24]. Furthermore, it has been illustrated by simulations that ACLMS yields suboptimal SI cancellation results in the presence of PA nonlinearity, since it arbitrarily excludes the nonlinear SI components from its underlying observation model, which in fact can be outstanding among all the imperfections, as shown in Section 2.3. Meanwhile, a theoretical quantification of this suboptimality is still missing. Therefore, in this section, we provide a comprehensive mean and mean square convergence analysis of the conventional ACLMS based SI canceller in the presence of both frequencydependent Tx and Rx I/Q imbalances and PA nonlinear distortion. A unified statistical framework to quantify the SI cancellation performance of ACLMS in both transient and steadystate stages is investigated. For rigor, the proposed analysis covers both the cases of low and high transmit powers. For the compactness of analysis, we first represent the widely linear model in (14) in an augmented form, given by
(20) 
where is the augmented SI vector, and is the augmented optimal endtoend system IRs, which model the transmit and receive frequencydependent I/Q imbalances, PA distortion and the residual of analog SI cancellation.
The ACLMS estimates the set of system parameters by minimising the MSE cost function , defined as [28, 29, 30]
(21) 
where is the instantaneous output error, given by
(22) 
in which the augmented weight vector of ACLMS, that is, , is updated as
(23) 
3.1 Mean Convergence Analysis
Upon introducing the weight error vector
(24) 
the filter output in (22) becomes
(25) 
From (23), the recursion for the update of the weight error vector can be derived as
(26) 
where is an identity matrix.
The mean behavior of can now be determined by applying the statistical expectation operator to both sides of (26) and upon employing the standard independence assumptions [40, 41, 42, 43, 44, 45], that is, the composite noise is statistically independent of any other signal in ACLMS, and is statistically independent of the augmented SI input vector , to yield
(27) 
where is the covariance matrix of the augmented SI vector , defined as
(28) 
Therefore, the convergence of ACLMS in the mean is guaranteed if the stepsize satisfies [44]
(29) 
From Fig. a and Fig. b in Section 2.3, we observe that the relative power relationships among different SI components vary as the transmit power changes. Therefore, in order to accurately describe the statistical mean behaviour of the ACLMS based SI canceller, we next consider two case studies as
Low transmit power
High transmit power
When the transmit power is high, the nonlinear PA distortion component becomes the third strongest interference among all the imperfections considered. Upon using the independence assumptions and (15), the second term on the RHS of (27) can be derived as
(31) 
where represents an zero vector. The last step is performed by using the Gaussian fourth order moment factorizing theorem and since is proper, we have [40, 41, 42, 43]. From (27), the steadystate value of the weight error vector, that is, , can be evaluated as
(32)  
Remark 1: The upper bound on the step size for the mean convergence of ACLMS for a low transmit power FD DCT is identical to that for a high transmit power one. At the steady state, when the transmit power is low, ACLMS converges in the mean to the optimal weight coefficients associated with , that is, in (20), in an unbiased manner. However, as indicated by (32), when the transmit power is high, this yields a bias in the estimation of out of entries of the weight error vector , quantified by . The level of this bias depends upon the level of undermodeling, that is, the transmitter mixer gain , the transmit SI signal power , and the channel IRs associated with the IMD SI components, that is, and , whose values are determined by receiver frequencydependent I/Q imbalances, PA memory and residual of analog SI cancellation, as shown in (10) and (11).
3.2 MeanSquare Convergence Analysis
From (21) and (25), and again by employing the standard independence assumptions stated in Section 3.1, the MSE of ACLMS based SI canceller, that is, , can be further evaluated as
(33)  
where is the covariance matrix of the augmented weight error vector . It can be observed from (33) that the mean square convergence analysis of ACLMS now rests upon both the first and second order statistical properties of . To this end, we first apply the Hermitian operator to both sides of (26), to yield
(34) 
Upon multiplying (34) to both sides of (26) with (34) and taking the statistical expectation , the evolution of the weight error covariance matrix now becomes
(35)  
where
(36) 
It can be observed that is independent from the IMD component , and hence [41, 42, 43]
(37) 
where is the pseudocovariance matrix of the augmented SI vector , given by
in which is an identity matrix. The term on the RHS of (37) can be decomposed as , where is a vector, whose entries are the diagonal elements of , defined as
(38) 
and denotes a vector in which all the entries are unities. Then, based on (35), the evolution of becomes
(39) 
The convergence of the recursion for the vector in (39) is subject to two conditions: 1) the terms , , and are bounded, which is guaranteed if is bounded; 2) all the eigenvalues of the transition matrix are less than unity [46]. From (29), Condition 1) holds when . Furthermore, the eigenvalues of , denoted by , where , can be obtained by solving , and from (39), it is easy to find that is Toeplitz, for which the diagonal elements are , and offdiagonal ones are . Hence, after a few manipulations, we have
(40)  
(41) 
Note that since , we have , and hence, Condition 2) is satisfied if , to yield
(42) 
Remark 2: The upper bound in (42) is tighter than that in Condition 1), and therefore, the mean square convergence of ACLMS based SI canceller in the presence of frequencydependent IQ imbalances and PA distortion is guaranteed if the stepsize satisfies (42).