Transceiver Design for Multi-user Multi-antenna Two-way Relay Cellular Systems

# Transceiver Design for Multi-user Multi- antenna Two-way Relay Cellular Systems

Can Sun,  Chenyang Yang,  Yonghui Li,  and Branka Vucetic,  Manuscript received May 30, 2011; revised February 16, 2012 and May 3, 2012.This work was supported in part by the International ST Cooperation Program of China (ISCP) under Grant No. 2008DFA12100, and in part by Australian Research Council (ARC) Discovery Projects DP120100190, DP110103324, DP0877090 and Linkage Projects LP0991663, and has been presented in part at the IEEE Global Telecommunications Conference (GLOBECOM), Miami, USA, December 2010, and at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.C. Sun and C. Yang are with the School of Electronics and Information Engineering, Beihang University, Beijing 100191, China (email: saga@ee.buaa.edu.cn, cyyang@buaa.edu.cn).Y. Li and B. Vucetic are with the School of Electrical and Information Engineering, University of Sydney, NSW 2006, Australia (email: {lyh,branka}@ee.usyd.edu.au).
###### Abstract

In this paper, we design interference free transceivers for multi-user two-way relay systems, where a multi-antenna base station (BS) simultaneously exchanges information with multiple single-antenna users via a multi-antenna amplify-and-forward relay station (RS). To offer a performance benchmark and provide useful insight into the transceiver structure, we employ alternating optimization to find optimal transceivers at the BS and RS that maximizes the bidirectional sum rate. We then propose a low complexity scheme, where the BS transceiver is the zero-forcing precoder and detector, and the RS transceiver is designed to balance the uplink and downlink sum rates. Simulation results demonstrate that the proposed scheme is superior to the existing zero forcing and signal alignment schemes, and the performance gap between the proposed scheme and the alternating optimization is minor.

{keywords}

Two-way relay, multi-user, multi-antenna, transceiver, cellular systems.

## I Introduction

Two-way relay (TWR) techniques have attracted considerable interest owing to its high spectral efficiency. Most of prior works study TWR systems with single user pair, where two users exchange information via a single relay station (RS) [1, 2, 3]. Various transmission schemes have been proposed for single antenna nodes [1] and multi-antenna nodes [2, 3].

Recently, the design for TWR systems is extended to multi-user cases [4, 5, 6, 7, 8, 9, 10, 11], which can be roughly divided into two categories based on the system topologies, i.e., symmetric and asymmetric systems. In symmetric systems [4, 5, 6], multiple user pairs exchange information via a RS. In asymmetric systems, a base station (BS) exchanges messages with multiple users [7, 8, 11, 9, 10], which is a typical scenario of cellular networks.

In this paper, we study multi-user TWR cellular system, where a multi-antenna BS communicates with multiple single-antenna users bidirectionally via a multi-antenna amplify-and-forward (AF) relay. Owing to the importance from practical perspective, there is a considerable amount of work on designing transceivers for such a system [8, 11, 9, 10]. However, its transceiver optimization is challenging due to the complicated interference among multiple users in the broadcast and multi-access phases, and even its bidirectional sum capacity is still not available until now.

Allocating orthogonal time or frequency resources to the uplink and downlink signals of different users is an immediate way to eliminate the interference [7], with which existing single-user TWR techniques can be directly applied. Since this is far from optimal, a further attempt is to introduce an interference free constraint, which is essentially the zero-forcing (ZF) principle. Though also suboptimal in a sense of sum rate, such a design can capture the inherent degrees of freedom of the system, which is an approximate characterization of the capacity at the high signal-to-noise (SNR) level. Along this line, several ZF-principle based transceivers have been proposed. Considering that the RS is equipped with multiple antennas, a natural solution is to apply ZF transceiver at the RS to separate all the signals from and to the BS and users [8]. This ZF scheme employs orthogonal spatial resources to differentiate different links, thereby the RS should be equipped with enough antennas. To remove all the interference, at least antennas are required at the RS for a system with antennas at the BS and single antenna users. When the RS is only with antennas, the multiple antennas at the BS also need to be exploited to ensure interference free transmission. In [9, 10, 11], the concept of signal alignment (SA) [12] is employed to reduce the number of interference experienced at the relay. The SA scheme exploits the self-interference cancelation (SIC) [13] ability of TWR. Its basic idea is to project the uplink and downlink signals of each user onto the same spatial direction at the RS through proper BS precoding, such that the RS can separate superimposed signals. After receiving a superimposed signal forwarded by the RS, each user removes its transmitted uplink signal via SIC, and obtains its desired downlink signal.

Both the ZF and SA schemes are based on ZF-principle. Nonetheless, they are not the only interference free solution111By using the terminology “interference free solutions”, we refer to the transmit strategies that can remove all interference. These solutions include the ZF beamforming and ZF detector, the SA scheme, as well as the transmit schemes using orthogonal frequency or time resources, which can null the interference thoroughly.. In fact, by analyzing the feasibility of interference free constraints for multi-user multi-antenna TWR cellular systems, it is not hard to show that the SA scheme is the unique solution only for special antenna configurations, and the ZF scheme ensures interference free transmission only when the number of antennas at the RS is sufficiently large. Moreover, both of them are designed as low complexity schemes without taking into account the sum rate.

In this paper, we strive to find a low complexity interference free transceiver towards maximizing sum rate under general antenna settings. To provide a performance benchmark as well as useful insight into the transceiver structure, we employ a standard alternating optimization technique [14] to optimize the BS and RS transceivers aiming at maximizing bidirectional sum rate under interference free constraints. In order to develop a low complexity transceiver scheme, we fix the BS transceiver as the optimal BS precoder and detector in high power region found from the alternating optimization. Based on which we first optimize the RS transceiver to separately maximize the uplink and downlink sum rates and then balance the uplink and downlink sum rates to maximize the bidirectional sum rate. Simulation results show that the balanced scheme performs very close to the alternating optimization solution, and outperforms existing ZF and SA schemes under various scenarios.

The rest of the paper is organized as follows. Section II describes the system model. Section III introduces the alternating optimization solution. The balanced transceiver scheme is proposed in IV. Simulation results are given in section V, and conclusions are drawn in section VI. The major symbols used in the paper are summarized in Table I.

## Ii System Model

We consider a multi-user multi-antenna TWR system, which consists of a BS equipped with antennas, a RS equipped with antennas and single-antenna users. The BS and multiple users exchange downlink and uplink information via the RS, as shown in Fig. 1. The bidirectional transmission takes place in two phases.

At the first phase, both the BS and multiple users transmit to the RS. The received signal at the RS is given by

 yr (1)

where is the channel matrix from the BS to the RS, , is the channel vector from the th user to the RS, and are the downlink and uplink signal vectors to and from users and we assume , is the transmit power of each user, is the Gaussian noise vector at the RS with zero mean and covariance matrix , and is the precoder matrix at the BS, which satisfies the transmit power constraint as follows

 ∥Wbt∥2≤PB, (2)

where is the maximal transmit power of the BS.

At the second phase, the RS precodes its received signals and then broadcasts them to the BS and users. The received signals at the BS and the th user are respectively given by

 yb =WTbr(HTbrWryr+nb), (3) yui =hTirWryr+nui,  (1≤i≤NU), (4)

where is the weighting matrix at the RS, is the receive weighting matrix at the BS, and and are Gaussian noises at the BS and the th user, each with zero mean and variance .

The RS weighting matrix should satisfy the transmit power constraint , which can be rewritten as follows after substituting (1),

 ∥WrHbrWbt∥2+PU∥WrHur∥2+N0∥Wr∥2≤PR, (5)

where is the maximal transmit power of the RS222We do not consider power control at the BS and RS. The inequality power constraints are for simplifying the optimization..

All channels are assumed independent quasi-static flat fading and we consider time division duplexing for simplicity, hence the channels in the 1st and 2nd phases are assumed reciprocal. We assume that the BS and RS have global channel information of all links as in [11, 9, 10].

## Iii Transceiver Design Based on Alternating Optimization

Even after we introduce the interference free constraints, the problem of jointly optimizing BS and RS transceivers that maximizes the bidirectional sum rate of multi-user multi-antenna TWR systems is still non-convex and is very hard to deal with. In this section, we employ a standard tool, alternating optimization [14], to solve the optimization problem, which can serve as a performance benchmark for the interference free transceivers.

Substituting (1) into (3), the received signal at the BS can be rewritten as

 yb= WTbrHTbrWrHbrWbtxb+√PUWTbrHTbrWrHurxu+ WTbrHTbrWrnr+WTbrnb, (6)

where the first term is the transmitted signal of the BS in the first phase which can be removed by SIC, the second term is the desired uplink signal, and the last two terms are the noise amplified by the RS and the noise at the BS receiver, respectively.

To eliminate the interference among the uplink signals, the following constraint should be satisfied,

 wTbriHTbrWrhjr=0, i≠j, (7)

where is the th column of .

Substituting (1) into (4), the received signals at the th user can be rewritten as

 yui= hTirWrHbrWbtxb+√PUhTirWrHurxu+hTirWrnr +nui   (1≤i≤NU), (8)

where the first term consists of the downlink signals for all users, the second term consists of the transmitted signals from users in the first phase, and the last two terms are noises.

To remove the interference, the BS and RS transceivers should satisfy the following constraints,

 hTirWrHbrwbtj=0, i≠j, (9) hTirWrhjr=0, i≠j, (10)

where is the th column of .

Considering the inter-user interference (IUI) free constraints (7), (9) and (10) and the fact that the self-interference can be canceled [13], the receive SNR of the th uplink and downlink signal can be respectively obtained as,

 SNRUi=PU|wTbriHTbrWrhir|2N0∥wTbriHTbrWr∥2+N0∥wTbri∥2, SNRDi=|hTirWrHbrwbti|2N0∥hTirWr∥2+N0. (11)

Then the bidirectional sum rate of the TWR system is333The received non-white noise after amplifying and forwarding is treated as white noise as in existing literature. This is in fact the worst case of the problem, therefore the data rate obtained by can serve as a lower bound.,

 RS =RU+RD=NU∑i=1(RUi+RDi) =NU∑i=1[12log2(1+SNRUi)+12log2(1+SNRDi)], (12)

where and denote the uplink and downlink sum rate, and are the uplink and downlink data rates of the th user, and the pre-log factor is due to the half-duplex constraint.

In the following, we optimize one of the three transceiver matrices by fixing the other two.

### Iii-a Optimization of Weighting Matrix Wr of RS

Here we fix and , and optimize to maximize the bidirectional sum rate under the RS transmit power constraint and the IUI-free constraints by solving the following problem,

 maxWr RS (13a) s.t. (???), (???), (???) and (???). (13b)

The bidirectional sum rate is not a convex function of . To solve this non-convex problem and find the maximum , we employ the concept of rate profile, which is introduced in [15] to characterize the boundary rate-tuples of a capacity region. We introduce a vector to specify the rate profile, where and . Then by solving the following optimization problem,

 maxWr RS (14a) s.t. RUi≥βiRS, RDi≥βi+NURS, 1≤i≤NU, (14b) (5), (7), (9) and (10), (14c)

we will achieve a boundary point of the achievable rate region specified by each vector . After searching the optimal from all its possible values, we can find the optimal boundary point corresponding to the maximum sum rate. For multi-user case, it is too complicated to search all possible . To reduce the complexity, we use bisection algorithm [16] to search the optimal as in [8]. Although it is hard to rigorously prove that the achievable rate region boundary is a convex hull in terms of , simulation results show that bisection algorithm offers the same result as that of using brute-force searching.

To solve the problem (14), we apply a similar approach as in [2] to convert the optimization problem (14) to a semidefinite programming (SDP) problem with a rank-1 constraint, and then we resort to the widely used semidefinite relaxation [17] to handle the problem.

### Iii-B Optimization of Transmit Weighting Matrix Wbt of BS

In this subsection, we design given and . Since the transmit weighting matrix of the BS only affects downlink rate when and are fixed, we design it to maximize the downlink sum rate . The design of should consider the IUI free constraint (9) and the BS transmit power constraint (2). It is also associated with the RS transmit power constraint (5). Then the optimization problem can be formulated as

 max Wbt RD (15a) s.t. (???), (???) and (???). (15b)

This is also a non-convex problem, which can be solved by the same method as that we used to solve problem (13). Define a vector , where and . The solution of (15) can be found from solving the following problem by searching the optimal ,

 max Wbt RD (16a) s.t. RDi≥βiRD,  1≤i≤NU, (16b) (???), (???) and (???). (16c)

Each of the BS and RS power constraints (2) and (5) imposes a constraint on the norm of a linear function of . The IUI free constraint (9) is a linear constraint on . According to [17], the rate tuple constraint (16b) can be converted to linear constraints on . Therefore the constraints in (16) form a second-order-cone feasible region [17], and the size of the feasible region depends on . Consequently, we can solve (16) by searching the maximal that guarantees a non-empty feasible region. Bisection method is applied to search . We use the CVX tool[18] to check whether the feasible region is empty or not. If it is not empty, the CVX tool will return a value of in the feasible region. Finally, we will obtain both the maximum value of and the optimal .

### Iii-C Optimization of Receive Weighting Matrix Wbr of BS

Given and , only affects uplink sum rate. Among the three IUI-free constraints, is only associate with (7). Therefore, the optimization problem can be formulated as

 maxWbr RU (17) s.t. wTbriHTbrWrhjr=0,  i≠j. (18)

According to (III) and (III), the data rate of each uplink stream, , is only a function of . Therefore, this problem can be decoupled into subproblems. Since is a monotonic increasing function of , each subproblem can be formulated as

 maxwbri SNRUi (19a) s.t. wTbriHTbrWrhjr=0,  j≠i. (19b)

Any feasible should satisfy , where is obtained from channel matrix with the th column being removed. Define as a matrix consisting of all the singular vectors of corresponding to its zero singular values. Then we have

 wbri =U⊥irx, (20)

where is an arbitrary vector.

Rewrite the expression of in (III) as follows,

 SNRUi= wTbri(PUHTbrWrhirhHirWHrH∗br)w∗briwTbri(N0HTbrWrWHrH∗br+N0INB)w∗bri ≜ wTbriKSw∗briwTbriKINw∗bri. (21)

where and .

By substituting (20) and (III-C), the optimization problem (19) becomes

 maxx (22)

which is a generalized Rayleigh ratio problem. The optimal is the eigenvector of corresponding to its largest eigenvalue [19].

By now, we have solved the three problems (13), (15) and (17). When we find the alternating optimization solution, we need to assign initial values for the transceiver matrices, which should satisfy all the IUI free constraints and the transmit power constraints. The initial values are set according to the following procedure.

First, constraint (10) can be rewritten as a group of linear equations of as , where denotes Kronecker product, and is the vectorization of a matrix by stacking its columns. The general solution of this equation is given by

 vec(Wr)=S⊥(KO)x, (23)

where is the matrix by stacking all , , is the orthogonal subspace of a matrix, and is an arbitrary vector.

We pick one from the general solution. Then we substitute the chosen into (7) and (9), find general solutions of these two set of equations similar to (23), and pick one and one among the general solutions. Finally, we multiply and with proper scalars to satisfy the BS and RS power constraints.

After assigning the initial values, we alternately optimize one of the three transceiver matrices by fixing the other two. The sum rate must increase with each iteration, otherwise, the iteration is terminated. Due to this requirement, the alternating procedure will surely converge. Because of the non-convex nature of the optimization problem, the converged solution is not guaranteed to be globally optimal, and depends on the initial values. Nevertheless, we can increase the probability to achieve the maximal bidirectional sum rate by repeating the alternating optimization procedure with multiple random initial values then picking the best solution.

## Iv Balanced Transceivers

In this section, we design a low complexity transceiver toward achieving maximal bidirectional sum rate under the interference free constraints. To this end, we decouple the joint optimization of the BS and RS transceivers resorting to the asymptotic analysis in high power region. Specifically, we first find the BS precoder and detector from analyzing asymptotic results of the alternating optimization solution. Then we optimize the RS transceiver based on the given BS transceiver, also in high power region.

### Iv-a BS Transceivers

#### Iv-A1 BS Precoder

To obtain a closed form solution, we consider an asymptotic region where the transmit power of RS goes to infinity. When , the RS transmit power constraint can be ignored, then the BS precoder optimization problem in (15) can be reformulated as

 max Wbt NU∑i=1log2(1+|hTirWrHbrwbti|2N0∥hTirWr∥2+N0) (24) s.t. ∥Wbt∥2≤PB, (25) hTirWrHbrwbtj=0, i≠j, (26)

which can be viewed as linear precoder optimization for downlink multi-user multi-antenna system with a channel matrix that maximizes the sum rate under interference free constraints and total transmit power constraint. According to [20], the optimal precoder is a ZF precoder with proper power allocation, i.e.,

 Wbt=(HTurWrHbr)†Gb, (27)

where is a diagonal power allocation matrix. For simplicity, we consider equal power allocation at the BS, i.e.,

 ∥wbti∥2=PB/NU. (28)

#### Iv-A2 BS Detector

To obtain a closed form detector, we consider another asymptotic region where the transmit power of the BS or users approaches infinity. When or , the received SNR at the RS in the first phase goes to infinity, then the RS forwarded noise can be neglected444This is not true for the case of deep fading where the channel coefficient is approximately zero, but such a case is of low probability.. In this case, in (22) is . By solving the problem (22) and applying (20), the optimal BS receiver vector can be obtained as , where spans the orthogonal subspace of [19]. Therefore, the optimal is the projection of onto the orthogonal subspace of , i.e., the optimal solution is a ZF receiver for the equivalent uplink channel , i.e.,

 Wbr=[(HTbrWrHur)†]T. (29)

Note that the obtained BS precoder and detector in (27) and (29) are not optimal for practical systems with finite transmit power. Nonetheless, later we will show by simulations that these ZF transceivers perform fairly well even when the transmit powers are finite.

### Iv-B RS Transceiver

Now we find the solution of RS transceiver from (13) given the BS transceivers (27) and (29). The IUI free constraints (7) and (9) are satisfied owing to the usage of ZF transceivers at the BS, and thus can be removed. Note that we consider equal power allocation in the BS precoder, then the optimization problem of RS transceiver can be reformulated as

 max  Wr RS (30a) s.t. (???), (???), (???), (???) and (???). (30b)

To find a low complexity solution for this non-convex problem, we decouple it into two subproblems, which respectively maximize the uplink and downlink sum rate. Then we combine these two solutions to maximize the bidirectional sum rate.

When the transmit power of each user goes to zero, i.e., 555This is not conflict with the optimality conditions of the ZF transceivers at the BS, which are and either or ,, the system uplink sum rate will approach to zero, then . From (III) and (III), the downlink sum rate does not depend on , therefore the constraint (29) in problem (30) can be removed. Moreover, in this case the RS received signal at the first phase . Then the RS power constraint (5) can be rewritten as . Consequently, the problem (30) reduces to the following problem that maximizes the downlink sum rate,

 max  Wr RD (31) s.t. (10), (27), (28) and ∥WrHbrWbt∥2+N0∥Wr∥2≤PR. (32)

Similarly, when the BS transmit power goes to zero, i.e., , the problem (30) reduces to the following problem that maximizes the uplink sum rate,

 max  Wr RU (33) s.t. (10), (29) and PU∥WrHur∥2+N0∥Wr∥2≤PR. (34)

We will first solve these two subproblems, then combine the two solutions of to balance the uplink and downlink rates, so as to maximize the bidirectional sum rate.

#### Iv-B1 Design of Wr From Subproblem (31)

We can show that the optimal solution of (31) has the following structure (see Appendix),

 Wr=(HTur)†Gr1UT, (35)

where is a diagonal matrix and each column of has unit norm, i.e., .

The optimal structure of can be intuitively explained as follows. When , there is only downlink transmission, i.e., the RS receives signals from the BS and then forwards it to the users. In this case, represents the ZF precoder at the RS to broadcast signals to the users, is a power allocation matrix for different signal streams, and is the receive weighting matrix at the RS, which separates the downlink signals from the BS, see Fig. 2.

To obtain the power allocation matrix , we simply let the amplification coefficients at the RS for all streams to be identical. Denote each column of as . Since each downlink data stream is received by , amplified by , and forwarded by , we design to ensure that each has the same norm.

Our next task is to design the receive weighting matrix . Upon substituting (35), the optimization problem (31) can be rewritten as

 max U 12NU∑i=1log2(1+p2ri|uTiHbrwbti|2N0p2ri+N0) (36a) s.t. uTihjr=0, i≠j,  ∥ui∥=1, (36b) Wbt=(UTHbr)†G−1r1Gb, (36c) ∥wbti∥2=PB/NU, (36d) ∥(HTur)†Gr1UTHbrWbt∥2+N0∥(HTur)†Gr1UT∥2 ≤PR. (36e)

This problem is non-convex, thereby we turn to find its suboptimal solution. Since acts as the receiver at the RS in downlink transmission, we design it to maximize the “data rate” of BS-RS transmission instead of the two-phase downlink transmission666Since the RS does not decode message in AF protocol, in fact there is no “BS-RS transmission data rate”. We use this terminology here for simplifying the optimization problem..

During the BS-RS transmission, the half-duplex RS only receives signals from the BS. Therefore, we do not consider the RS transmit power constraint (36), which can be met later by adjusting . Then the BS-RS transmission rate maximization problem is formulated as

 max U NU∑i=1log2(1+|uTiHbrwbti|2/N0) (37a) s.t. (36b), (36c) and (36d). (37b)

Remark 1: If , the objective function (36a) will be the same as (37a) except for the pre-log factor , and the RS power constraint (36) can be omitted. This means that the two optimization problems are approximately equivalent when the RS has high transmit power.

Constraint (36c) shows that is a pseudo inverse of with power allocation. Define as the matrix with the th column being removed. Then using the principle of orthogonal projection[19], we obtain that

 |uTiHbrwbti|/∥wbti∥ = ∥uTiHbr(I−HHbr¯¯¯¯¯U∗i(¯¯¯¯¯UTiHbrHHbr¯¯¯¯¯U∗i)−1¯¯¯¯¯UTiHbr)∥ ≜ ∥uTiHbrS⊥(¯¯¯¯¯UTiHbr)∥.

Substituting this expression and (36d) into (37a), then the problem (37) can be rewritten as

 max U NU∑i=1log2(1+PBNUN0∥uTiHbrS⊥(¯¯¯¯¯UTiHbr)∥2) s.t. uTihjr=0, i≠j,  ∥ui∥=1. (38)

Constraints (36c) and (36d) are omitted since the objective function does not rely on now.

Solving problem (IV-B1) is nontrivial because we need to jointly design all . To obtain a low-complexity solution, we employ alternating optimization [14] again. We first initialize . Then we alternately optimize each of the columns of . In each step, we optimize the th column by solving the problem (IV-B1) with all other columns being fixed. After each step, we renew the matrix by replacing its th column by the optimized . The procedure stops when the value of objective function in (IV-B1) does not increase any more. Simulations show that the procedure always converges after each of the columns has been optimized once.

In the above procedure, we need to solve the optimization problem (IV-B1) with fixed . Note that the constraint , can be rewritten as . Any feasible must lie in the orthogonal subspace of . Therefore, we have

 ui=S⊥(¯¯¯¯¯Hir)x, (39)

where is an arbitrary vector. Then the optimization problem (IV-B1) with fixed can be rewritten as follows by substituting (39),

 max x ∥xTS⊥(¯¯¯¯¯Hir)THbrS⊥(¯¯¯¯¯UTiHbr)∥ s.t. ∥x∥=1. (40)

The optimal value of is the left singular vector of corresponding to its largest singular value [19]. Then from (39), we can obtain the optimal .

Substituting the optimization result into (35), the RS weighting matrix designed for maximizing the downlink sum rate can be obtained as .

#### Iv-B2 Design of Wr From Subproblem (33)

We can also show that the optimal RS weighting matrix that maximizes the uplink sum rate has the following structure,

 (41)

where and can be obtained similarly as in the last subsection. We do not present the detailed derivation for concision.

#### Iv-B3 Balancing Ru and Rd to Maximize Bidirectional Sum Rate

Consider that bidirectional sum rate , while and are respectively optimized for and . To improve , we propose the following RS weighting matrix,

 WBLr=cγ(γW⋆1r+(1−γ)W⋆2r), (42)

where the power adjusting factor , , is used for controlling the power proportion to and to balance the uplink and downlink sum rate, is used to meet the total RS transmit power constraint. In practical systems, after obtaining and , the RS can search for an optimal that maximizes the bidirectional sum rate.

Remark 2: In a TWR system with single user and single-antenna BS, turns out to be a maximal-ratio combination and maximal-ratio transmission (MRC-MRT) weighting matrix. It is shown in [2] that the bidirectional sum rate gap between MRC-MRT and the optimal scheme is no more than 0.2bps/Hz. Though in multi-user multi-antenna TWR system, we cannot draw the same conclusion via rigorously analysis, the simulation results in section V will show that such a balanced solution performs closely to the alternating optimization solution.

Remark 3: By substituting and into (42), we have

 WBLr=cγ(γ(HTur)†Gr1U⋆T+(1−γ)U⋆Gr2H†ur). (43)

As shown in (39), each column of matrix lies in the orthogonal subspace of , i.e., . The th column of also lies in that orthogonal subspace, i.e., .

When , i.e., the number of RS antennas equals to the number of users, the matrix is a matrix. Therefore, the rank of its orthogonal subspace is one. Since both of and lie in the same rank-1 subspace, they are linearly dependent, i.e., , where is a scalar. Therefore, we have , where . Substituting this expression into (43), we obtain

 WBLr =(HTur)†(cγγGr1DT+cγ(1−γ)DGr2)H†ur ≜(HTur)†GBLrH†ur, (44)

where is a diagonal matrix.

Comparing (IV-B3) with the RS transceiver in the SA scheme proposed in [11, 9, 10], we see that has the same form as that of the SA scheme. Substituting (IV-B3) into (27) and (29), it is easy to show that the BS transceivers in our balanced solution also have the same forms as those in the SA scheme. This indicates that the SA scheme is a special case of the balanced solution when . In fact, in such a setting, it is not hard to show that the solution of interference free constraints (7), (9) and (10) is unique, which is exactly the SA scheme.

### Iv-C Complexity Comparison

Here we compare the computational complexities of the balanced scheme and the existing ZF [8] and SA schemes [9, 10, 11].

In the ZF scheme, the major operation at the RS is to compute the pseudo inverse of a matrix. Since all the interference are eliminated by the RS, the BS needs to do nothing. In the SA scheme, the major operations at the RS and the BS are to compute the pseudo inverses of a matrix and a matrix, respectively.

In the balanced scheme, to obtain the RS transceiver (42), we need to compute , , and , and search for the optimal balancing factor . Specifically, we need to perform times of singular vector decomposition (SVD) to alternately design the columns of . According to (IV-B1), each SVD is performed for a matrix. Only vector norm operation is required to compute the power allocation matrices and , for which the complexity can be neglected compared with those of the pseudo inverse and SVD. The complexity of finding the optimal can also be ignored, which only requires a scalar searching operation. To obtain the BS transceivers in our balanced scheme (27) and (29), we need to compute the pseudo inverses of two matrices.

A widely used method for computing pseudo inverse is using SVD, which results in a complexity of flops to compute the pseudo inverse of a matrix [21], where . The complexities of the transceiver schemes are present in Table II, which shows that the complexity of the balanced scheme is on the same order as those of the SA and ZF schemes.

## V Simulation Results

In this section, we evaluate the performance of the proposed transceivers and compare them with existing schemes by simulations. We assume that all channels are independent and identically distributed Rayleigh fading channels, and all simulation results are obtained by averaging over 1000 Monte-Carlo trails. For a fair comparison, we use equal power allocation at the BS and RS in all the transceiver schemes. We assume that the noise variance is identical at the BS, RS and each user. The transmit power of each user . The BS and RS transmit power are and , respectively. We define as the transmit SNR. Without otherwise specified, we set , , , , and dB.

### V-B Convergence of the Alternating Optimization Solution

To study the convergence of the alternating optimization algorithm, we respectively use the proposed balanced transceiver, the ZF and SA transceivers and multiple random weighting matrices as its initial value. When using random matrices as the initial values, we pick one from multiple results that converges to the highest sum rate.

Figure 4 shows the bidirectional sum rate versus the iteration number. The sum rate converges rapidly but the converged result depends on the initial values due to the non-convexity nature of the optimization problem. Nonetheless, by using multiple random initial values, higher bidirectional sum rate can be achieved. We observe from extensive simulations that when the number of random initial values exceeds 20, the performance gain is marginal. Therefore, we can take the result with 20 random initial values as a near-optimal result. It is shown that the performance of the balanced transceiver is very close to that of the near-optimal result. In the following, we will use the balanced transceiver as the initial value for the alternating optimization.

### V-C Comparison among Different Transceivers

We compare the bidirectional sum rates of alternating optimization solution and the balanced transceiver with those of the ZF [8] and SA schemes [11, 9, 10]. We also compare with a minimum-mean-square-error (MMSE) transceiver without the interference free constraints, where the MMSE BS transceiver and MMSE RS transceiver were alternately optimized [22].

Figure 5 shows the impact of the antenna number of the RS, where “Al-Opt” denotes the alternating optimization solution. When there are two users, the ZF scheme needs at least 4 antennas at the RS to cancel all the interference, while the SA scheme only needs 2 antennas. From the simulation results, we see that when the sum rate of the ZF scheme reduces sharply due to the residual IUI, but the SA scheme performs much better. When , the ZF scheme becomes superior because it can remove all IUI but the SA scheme suffers from a power loss when aligning the downlink signals with the uplink signals. The sum rate of the balanced transceiver is close to that of the alternating optimization solution, both are higher than the existing ZF and SA schemes for any antenna number at the RS.

Figure 6 shows the impact of the user number on the performance of different transceivers. We set , and . Round robin scheduler is applied, where the scheduled user number is from 1 to 4. It shows that the performance of the ZF scheme degrades severely when because the four-antenna RS can not cancel all IUI. With the SA scheme, the proposed balanced scheme and the alternating optimization solution, the system achieves the highest bidirectional sum rate when three users are scheduled, where both the balanced scheme and alternating optimization result have about 2bps/Hz sum rate gain over the SA scheme. When , we see that the performance of the balanced transceiver and the SA scheme are exactly the same. This agrees well with our earlier analysis in Remark 3.

In Fig. 7 we compare the sum rate of the interference free transceiver schemes with the MMSE transceiver [22]. We can see that our balanced scheme provides higher sum rate than the existing ZF and SA schemes in a wide range of transmit SNR. The MMSE scheme is slightly superior to our balanced scheme in low SNR region, but is inferior to the proposed scheme in high SNR region. This is because the MMSE solution in [22] is obtained via alternating optimization, which is not guaranteed to be globally optimal. In high SNR region, the system is interference-limited, therefore the proposed scheme outperforms the MMSE solution by removing all the interference.

In Fig. 8, we provide the sum rate under each single channel realization to understand the behavior of the IUI free transceivers. We see that the ZF and SA schemes perform differently for a given channel. The ZF scheme requires the RS to separate all the signals transmitted by the users and BS, and performs well only when the channel vectors from the users and the BS are mutually orthogonal. Contrarily, the SA scheme needs to align the signals transmitted by the BS onto the same directions of the signals transmitted by the users, and thus performs well only when the channel vectors from the users and those from the BS have the same direction. Our balanced scheme can adaptively adjust transmission strategy depending on the channel condition to ensure IUI free without the requirements for channel “orthogonalization” or “alignment”. Therefore, its sum rate is always higher than those of ZF and SA schemes.

In Fig. 9, we compare the outage probabilities of the IUI free transceivers with Monte-Carlo trails, where the system is in outage if its bidirectional sum rate drops below a given threshold, which is set as 2bps/Hz. We see that our balanced scheme achieves much lower outage probability than both the ZF and SA schemes. Moreover, since the sum rate of the balanced scheme is always “riding on the peak” of the ZF and SA schemes as shown in Fig. 8, the balanced scheme achieves higher diversity gain, and its outage probability decreases much faster than those of the ZF and SA schemes as the SNR increases.

## Vi Conclusion

In this paper, we have designed transceiver for multi-user multi-antenna two-way relay systems. We first employed alternating optimization to find the BS and RS transceivers that maximizes the bidirectional sum rate under interference free constraints. We proceeded to propose a low complexity balanced transceiver scheme. By analyzing the solution of the alternating optimization in high transmit power region, we find that zero-forcing BS transceivers are asymptotically optimal. Given the BS transceivers, we designed the RS transceivers by respectively maximizing the uplink and downlink rate, which are then combined with a power adjustment factor to maximize the bidirectional sum rate. Existing signal alignment scheme was shown as a special case of the balance scheme where the relay antenna number equals to the user number. Simulation results showed that the performance gap between the balanced scheme and the alternating optimization solution is minor. In general system settings, the bidirectional sum rate of the balanced transceiver is higher than the existing signal alignment and zero-forcing schemes.

### Proof of the Optimal Structure of Wr in (35)

Define as a matrix consisting of the singular vectors of , and as a matrix consisting of the singular vectors of the orthogonal subspace of . Then is a unitary matrix, and can be expressed as

 Wr=VFVHFWr=VurAT+V⊥urBT, (45)

where , are two arbitrary matrices.

Since