Joint Beamforming and Power Allocation in Downlink NOMA Multiuser MIMO Networks

# Joint Beamforming and Power Allocation in Downlink NOMA Multiuser MIMO Networks

Xiaofang Sun,  Nan Yang,
Shihao Yan,  Zhiguo Ding,
Derrick Wing Kwan Ng,  Chao Shen,
and Zhangdui Zhong,
X. Sun, C. Shen, and Z. Zhong are with the State Key Lab of Rail Traffic Control and Safety and the Beijing Engineering Research Center of High-speed Railway Broadband Mobile Communications, Beijing Jiaotong University, Beijing 100044, China (emails: {xiaofangsun, chaoshen, zhdzhong}@bjtu.edu.cn). X. Sun and N. Yang are with the Research School of Engineering, Australian National University, Canberra, ACT 2601, Australia (emails: {u1029584, nan.yang}@anu.edu.au). S. Yan is with the School of Engineering, Macquarie University, Sydney, NSW, Australia (email: shihao.yan@mq.edu.au). Z. Ding is with the School of Electrical and Electronic Engineering, University of Manchester, Manchester, UK (email: Zhiguo.ding@gmail.com). D. W. K. Ng is with the School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia (email: w.k.ng@unsw.edu.au).
###### Abstract

In this paper, a novel joint design of beamforming and power allocation is proposed for a multi-cell multiuser multiple-input multiple-output (MIMO) non-orthogonal multiple access (NOMA) network. In this network, base stations (BSs) adopt coordinated multipoint (CoMP) for downlink transmission. We study a new scenario where the users are divided into two groups according to their quality-of-service (QoS) requirements, rather than their channel qualities as investigated in the literature. Our proposed joint design aims to maximize the sum-rate of the users in one group with the best-effort while guaranteeing the minimum required target rates of the users in the other group. The joint design is formulated as a non-convex NP-hard problem. To make the problem tractable, a series of transformations are adopted to simplify the design problem. Then, an iterative suboptimal resource allocation algorithm based on successive convex approximation is proposed. In each iteration, a rank-constrained optimization problem is solved optimally via semidefinite program relaxation. Numerical results reveal that the proposed scheme offers significant sum-rate gains compared to the existing schemes and converges fast to a suboptimal solution.

NOMA, MIMO, CoMP, beamforming, power allocation, successive convex approximation.

## I Introduction

Non-orthogonal multiple access (NOMA) has recently attracted much attention in both industry and academia, as a promising technique for providing superior spectral efficiency in 5G wireless networks [2]. Specifically, NOMA is a multiuser multiplexing scheme which enables simultaneous multiple access in the power domain [3]. This makes it fundamentally different from conventional orthogonal multiple access (OMA) schemes, such as time division multiple access, frequency division multiple access, and code division multiple access. Aided by NOMA, a base station (BS) is able to serve multiple users at the same time, frequency, and spreading code but at different power levels yielding a higher flexibility and a more efficient use of spectrum and energy. In order to unlock the potential benefit of NOMA, successive interference cancellation (SIC) is normally performed at some users such that they can remove the co-channel interference incurred by NOMA and decode the desired signals successively [4]. In practice, NOMA allocates more power to the users with poor channel qualities to ensure the achievable target rates at these users, thus striking a balance between network throughput and user fairness [5, 4]. Moreover, NOMA has a definite superiority over OMA in terms of sum channel capacity and ergodic sum capacity [6].

### I-a Related Studies and Motivations

Motivated by the fundamental works which established the concepts of NOMA (see [7, 8, 9, 10] and the references therein), [11] and [12] systematically evaluated the performance of NOMA in the downlink and the uplink, respectively. To maximize the energy efficiency for a downlink multi-carrier NOMA system, [13] optimized subchannel assignment and power allocation. Advocated by the unique benefit of multi-antenna systems, the application of multiple-input multiple-output (MIMO) techniques to NOMA was addressed in [6, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. For instance, [6] adopted an identity matrix as the precoder and assumed that users are equipped with more antennas than the BS; thus users applied zero-forcing (ZF) approach to eliminate intra-cluster interference. [17] proposed a minorization-maximization based algorithm to maximize the downlink sum-rate, where the transmit signals of each user are processed by a sophisticated precoding vector. Considering a multiuser system where users transmiting multiple data streams, [18] solved a beamforming power minimization problem by firstly obtaining the optimal power allocation for given beamforming vectors and then finding the optimal beamforming vectors iteratively. [19] considered the application of NOMA to a multi-user network with mixed multicast and unicast traffic. [20] aimed to maximize the system throughput and adopted an iteratively weighted minimum mean square error approach to design the beamformer. Recently, the authors in [21] proposed a user clustering scheme and adopted the ZF beamforming approach for the maximization of the throughput in a single-cell scenario. Specifically, [22] introduced simultaneous wireless information and power transfer into NOMA systems focusing on a two-user multi-antenna single-cell scenario. As an enhanced version of conventional MIMO, [23, 24] proposed massive-MIMO-NOMA downlink transmission protocols. In particularly, the previous works relied on a key assumption that users have significantly different channel gains. However, in some practical situations, e.g., the Internet-of-Things (IoT) scenarios [25], the locations of users are close to each other and hence pairing users in terms of channel gains cannot fully exploit the promised performance gain from NOMA. To expand the limited application of NOMA, in this work, we study the joint beamforming and power allocation design where users have similar channel gains to further unlock the potential benefit of NOMA.

We note that [6, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] focused on the application of NOMA in single-cell multi-antenna scenarios. Nevertheless, the spectral and energy efficiencies can be further improved by introducing NOMA into multi-cell systems, which has also been investigated in [26, 27]. As shown in [26, 27], interference is a key limiting factor in improving the capacity of multi-cell networks. To address this issue, multi-cell cooperation has been proposed in [28]. Among various multi-cell cooperation techniques, coordinated multipoint (CoMP) is a promising and appealing one as it enables an adaptive coordination among multiple BSs [29]. The ultimate goal of CoMP is to enhance the quality of useful signals and to mitigate the undesired interference for improving the network efficiency and providing high quality-of-service (QoS) to users, especially for the users, e.g. at the cell-edge, suffering from poor channel qualities. To this end, multiple BSs can adopt either the coordinated scheduling/beamforming scheme or the joint processing CoMP scheme to facilitate cooperation [30]. For the former scheme, the data of a user is required to be available only at its associated BS, but not to other BSs. Yet, user scheduling and beamforming decisions are made jointly via the coordination among the BSs in the network. Differently, for the latter scheme, user data is shared among multiple BSs of the network, thus requiring backhaul links with extremely high capacity for information exchanging between all the BSs. In this paper, we focus on the former scheme to design the beamforming, as coordinated beamforming offers promising performance gains via interference avoidance and is less sophisticated compared to joint transmission [31].

### I-B Contributions

In this paper, we propose a new joint beamforming and power allocation design for a generalized multi-cell MIMO-NOMA network. Notably, we study a new scenario where the users are divided into two groups based on their QoS requirements. Specifically, the users in Group 1 are expected to be served with the best-effort, while the users in Group 2 impose strict QoS requirements and need to be served with their required target rates. Moreover, we consider a multi-cell network in this paper, which is different from the single-cell system considered in our previous study [1]. In this multi-cell network, our proposed design allows the multiple BSs to cooperate with each other and jointly design beamforming vectors and power allocation coefficients, which effectively suppresses the inter-cell interference. In particular, the proposed beamforming design takes into account the heterogeneous QoS requirements of users and ensures a sufficient disparity of the effective channel gains between any paired users. Therefore, the advantage of NOMA can be exploited.

The main contributions of this paper are summarized as follows:

• To jointly design beamforming and power allocation in the multi-cell multiuser MIMO-NOMA network, we first formulate a joint design problem to maximize the sum-rate of users in Group 1, while ensuring the target rates at the users in Group 2. Since this problem formulation is non-convex in general and hence challenging to solve, we propose a series of transformations to simplify the problem. We then propose an iterative suboptimal algorithm based on the successive convex approximation (SCA) to perform the coordination among BSs for joint design of beamforming vectors and power allocation coefficients.

• Unlike the problems studied in the literature which can be directly solved by semidefinite relaxation (SDR), e.g. [32, 33, 34], the use of SDR in this work leads to non-convex quadratic constraints and the proof of rank-one solution is non-trivial. However, we analytically prove that the SDR is tight and verify our finding via simulation. As such, the semidefinite program (SDP)-relaxed problem is equivalent to the original joint design problem.

• Through numerical results, we demonstrate that our proposed design outperforms the existing NOMA and OMA schemes. Notably, we find that our proposed design can always guarantee the QoS requirements of the users in Group 2, while the existing NOMA scheme cannot. We also investigate the impact of network parameters, such as the maximum transmit power and the numbers of cells, on the performance of our proposed design compared with existing schemes. We further find that our proposed iterative resource allocation algorithm quickly converges to a suboptimal solution, i.e., in no more than iterations on average, and examine the impact of various network parameters on the convergence rate. Moreover, we have confirmed that our scheme exhibits a comparable performance to the optimal solution produced by the brute-force method in a small scale network, but incurring a much lower complexity.

The rest of the paper is organized as follows: Section II presents the system model and Section III formulates the resource allocation design as an optimization problem. In Section IV, the SDR approach and the SCA-based iterative algorithm are proposed to solve the joint beamforming and power allocation design problem. Simulation results are given in Section V. Finally, Section VI concludes the paper.

Notation: Vectors and matrices are denoted by lower-case and upper-case boldface symbols, respectively. denotes the Hermitian transpose. denotes the trace operation. and denote the rank and the null space of , respectively, denotes the Euclidean norm, and denotes the absolute value. The distribution of a circularly symmetric complex Gaussian (CSCG) variable with mean and covariance is denoted by , and means “distributed as”. denotes statistical expectation. denotes the first partial derivative of function with respect to variable .

## Ii System Model

In this paper, we consider an -cell multiuser MIMO network, as shown in Fig. 1. In each cell, a BS equipped with antennas communicates with single-antenna users, where we assume that . The cell region of each BS is modeled as a disc with radius , where the BS is located at the center of the disc. The users in each cell are assumed to be divided into two groups, namely, Group 1 and Group 2. Without loss of generality, we assume that the number of users in Group 1 is the same as that in Group 2 in each cell, i.e., users in Group 1 and users in Group 2. In the considered network, NOMA is applied at the BSs such that they are able to simultaneously serve the two groups of users imposing different QoS requirements. We assume that every two users, one from Group 1 and one from Group 2, share the same resource (e.g. time slot, frequency, and space). These two users are considered as a cluster. The index of the -th cluster served by the -th BS is denoted by , where and . The user in Group 1 and the user in Group 2 in cluster are denoted by and , respectively. We introduce the set to represent all the BSs in this network except for the -th BS and denote the set of the indices of the BSs in by , i.e., . Aiming at studying the beamforming design for pairing users with heterogeneous QoS requirements and taking fully advantage of CoMP, we assume that perfect channel state information (CSI) of all the users in the network, i.e., the full global CSI, is available at each BS via channel reciprocity in time-division duplex systems or users feeding back in frequency-division duplex systems. The global CSI can be obtained based on the channel estimation methods studied in the literature, e.g., [35, 36, 37].

In practical scenarios, the disparity between the channel conditions of two users is not necessarily significant, which brings difficulties to user pairing for implementing NOMA. Motivated by this, this work considers the scenario where the QoS requirements of the users in two groups are significantly different. For instance, the users in Group 1 requiring non-delay sensitive applications are expected to be served with the best-effort, whereas the users in Group 2 request delay sensitive applications requiring a constant data rate.

## Iii Cooperative Transmission with NOMA

### Iii-a Zero Forcing Transmission at BSs

The information bearing vector adopted at BS , denoted by , is given by , where is the signal for cluster , and are the signals intended to and , respectively, and and are the portion of the transmit power allocated to and , respectively. Without loss of generality, we assume that to guarantee the total transmit power constraint.

To facilitate the downlink multiuser transmission, an beamforming matrix is adopted at BS . Mathematically, can be expressed as , where is the beamforming vector designed for cluster . As observed in [3], a significant performance gain of NOMA over conventional OMA is achieved in the high signal-to-noise ratio (SNR) regime, particularly when the channel qualities of the two users served using NOMA are significantly different from each other. Motivated by this observation, to enlarge the disparity between the received SNRs of the two users in one cluster, in this work, we adopt ZF beamforming [38] to eliminate the intra-cell interference at from other clusters within the same cell. This is due to our assumption that is expected to be served with the best-effort, while only requires a constant target data rate. We note that the ZF beamforming only requires local CSI at BS , which reduces the signaling overhead for cooperation. We also note that the QoS requirement at has to be guaranteed. Thus, it is important to ensure that the received interference at is carefully controlled. To this end, the beamforming vector, , needs to be designed such that receives a tolerable interference to achieve the target rate. To fulfill the aforementioned requirements, we rewrite the beamforming vector as , where is designed to eliminate the intra-cell interference for and aims to improve the data rate of while guaranteeing the QoS requirement of .

For the sake of clarity, we define a set of new matrices , which contains the channel vectors from BS to the remaining associated users in Group 1 except for . As such, we have , where represents the channel vector from BS to . Applying the singular value decomposition (SVD), we rewrite as , where is the first left eigenvectors of forming an orthogonal basis of and is the last left eigenvectors of (corresponding to zero eigenvalues) forming an orthogonal basis of the null space of . Since is used to eliminate the intra-cell interference for , it lies in the null space of and can be written as without loss of generality. Therefore, the signal transmitted by BS is given by

 xn=Pnsn=K∑k=1Uknqkn(√akns1kn+√bkns2kn). (1)

### Iii-B SIC at Users in Group 1

The received signal at is given by

 y1kn=gHnknxn+∑i∈¯¯¯¯¯NngHiknxi+ωkn, (2)

where denotes the additive white Gaussian noise (AWGN) at with zero mean and variance . Since ZF beamforming is adopted at BS , we have . Thus, substituting (1) into (2) yields

 y1kn= √akngHnknUknqkns1kndesired signal+√bkngHnknUknqkns2kn% intra-cluster interference+ ∑i∈¯¯¯¯¯NnK∑j=1gHiknUjiqjisjiinter-cell % interference+ωkn. (3)

In (III-B), the first term on the right-hand side is the desired signal for , the second term is the intra-cluster interference, i.e., the interference caused by the signal intended to , and the third term is the inter-cell interference caused by BSs in .

In the considered network, we assume that SIC is adopted at to remove the interference caused by , due to the required best-effort service for and a low target data rate requirement at . Hence, has to decode first. Based on (III-B), the signal-to-interference-plus-noise ratio (SINR) for decoding at is given by

 SINR2k1n= bkn|gHnknUknqkn|2akn|gHnknUknqkn|2+∑i∈¯¯¯¯¯Nn∑Kj=1|gHiknUjiqji|2+σ2kn, (4)

where we treat the interference as noise as commonly adopted in the literature, e.g.  [11]. Without loss of generality, we assume that the required target data rates at users in Group 2 are the same, which is defined by . In order to guarantee the success of SIC, we need to ensure . After performing SIC, the SINR for at user is given by

 SINR1k1n=akn|gHnknUknqkn|2∑i∈¯¯¯¯¯Nn∑Kj=1|gHiknUjiqji|2+σ2kn. (5)

### Iii-C Direct Decoding at Users in Group 2

We next study the received signal at , i.e., the user from Group 2 in cluster . We note that it is impossible to completely cancel the intra-cluster interference, the intra-cell interference, and the inter-cell interference at . As such, the received signal at is given by

 y2kn=√bknhHnknUknqkns2kn+Ikn+ϖkn, (6)

where represents the channel vector between BS and . Variable is the AWGN at . In (6), the first term on the right-hand side is the desired signal for and represents the interference. Here, is given by

 Ikn= √aknhHnknUknqkns1knintra-cluster interference+K∑j=1,j≠khHnknUjnqjnsjnintra-cell interference +∑i∈¯¯¯¯¯NnK∑j=1hHiknUjiqjisjiinter-cell % interference, (7)

where the first term on the right-hand side is the intra-cluster interference, the second term is the intra-cell interference, and the third term is the inter-cell interference caused by the BSs in . Following (6), the SINR for at is given by

 SINR2k2n=bkn|hHnknUknqkn|2E[|Ikn|2]+ς2kn, (8)

where the interference power is given by

 E[|Ikn|2]= akn|hHnknUknqkn|2+K∑j=1,j≠k|hHnknUjnqjn|2 +∑i∈¯¯¯¯¯NnK∑j=1|hHiknUjiqji|2. (9)

Note that in the considered network, we need to guarantee the QoS requirement at . As such, we need to ensure via a careful design of resource allocation.

### Iii-D Problem Formulation

In the considered system, the BSs need to maximize the sum data rates of the users in Group 1 while guaranteeing the target rates required at the users in Group 2. To this end, the BSs need to optimally design the beamforming vectors and the power allocation coefficients, i.e., and . Therefore, the optimization problem for the BSs, denoted by P1, is formulated as

 P1: maximize{akn,bkn,qkn}∀n,k N∑n=1K∑k=1log2(1+SINR1k1n) (10a) s.t. log2(1+SINR2k1n)≥R0,∀n,k, (10b) log2(1+SINR2k2n)≥R0,∀n,k, (10c) K∑k=1Tr(qknqHkn)≤Pn,∀n, (10d) akn+bkn=1,∀n,k, (10e) akn≥0,bkn≥0,∀n,k, (10f)

where is the data rate for users in Group 2. As per the Shannon’s coding theorem, the data rates are required to be less than the channel capacities to correctly decode the data bits at the receivers. Hence, constraint (10b) ensures the success of SIC decoding at , constraint (10c) guarantees the required target data rate at , and constraint (10d) imposes the maximum total power budget, , , at each BS. We also note that the beamforming vectors, i.e., , determine the power allocation among clusters at each BS, and the power allocation constraints for and in each cluster are given by (10e) and (10f). We further note that the objective function given by (10a) and the constraints given by (10b) and (10c) are non-convex with respect to , , and , due to the coupling between optimization variables in the objective function and constraints (10a), (10b), and (10c).

## Iv Joint Design of Beamforming and Power Allocation

In this section, we aim to solve P1 in (10). To this end, a series of transformations are proposed to simplify this problem. Then, SDR [32] and SCA [39] approaches are applied to perform the joint design of beamforming vectors and power allocation coefficients.

### Iv-a Problem Reformulation

P1 is a non-convex problem which belongs to the class of NP-hard problems [32]. In general, a brute-force approach is needed to obtain a globally optimal solution. Thus, in this section, we first transform P1 into an equivalent rank-constrained SDP problem to facilitate the design of a computationally efficient resource allocation. Specifically, we find that the beamforming variable in (10) is in the form of , , . Inspired by this, we introduce and optimize the auxiliary optimization matrix , , . It is noted that is required to be a rank-one positive semidefinite (PSD) matrix, i.e., and . Then, P1 can be equivalently written as P2 in terms of which is given by

 P2: maximize{akn,Qkn}∀n,∀k N∑n=1K∑k=1log(1+aknTr(GknknQkn)wkn) (11a) s.t. aknTr(GknknQkn)≤ Tr(GknknQkn)1+γ−γ1+γwkn,∀n,k, (11b) aknTr(HknknQkn)≤ Tr(HknknQkn)1+γ−γ1+γrkn,∀n,k, (11c) K∑k=1Tr(Qkn)≤Pn,∀n, (11d) Qkn⪰0,∀n,k, (11e) 0≤akn≤1,∀n,k, (11f) Rank(Qkn)≤1, (11g)

where ,

 Gjikn=UjigikngHiknUjiσ2kn,  Hjikn=UjihiknhHiknUjiς2kn, wkn=∑i∈¯¯¯¯¯NnK∑j=1Tr(GjiknQji)+1, rkn=K∑j=1,j≠kTr(HjnknQjn)+∑i∈¯¯¯¯¯NnK∑j=1Tr(HjiknQji)+1.

The proposed change of variables enables us to transform the considered problem with respect to to a rank-constrained SDP problem with respect to . We note that the optimization problem P2 is equivalent to the original problem P1 if and only if is a rank-one PSD matrix. If the rank-one constraint is guaranteed, the vector solution to P1 can be retrieved from the matrix solution to P2. On the other hand, even if the rank-one constraint on is dropped, problem P2 is still intractable due to the coupling between and . In the following, we will further transform and approximate problem P2 to obtain a tractable formulation.

Now, we handle the coupling between the optimization variables in the objective function. We note that the logarithm function in the objective function is concave with respect to the input argument. However, due to the received inter-cell interference involved in the denominator and the joint design of beamforming vectors and power allocation in (11a), the objective function is non-convex with respect to and . As such, we first adopt the following transformation to the objective function to circumvent its non-convexity.

We introduce a set of auxiliary variables to bound from below, i.e., the achievable SINR at users in Group 1. Specifically, is given by

 ρkn≤aknTr(GknknQkn)wkn. (12)

Substituting into (11a), P2 is transformed into an equivalent optimization problem P2a, which is given by

 P2a:  maximize{akn,Qkn,ρkn}∀n,k N∑n=1K∑k=1log2(1+ρkn) (13a) s.t. aknTr(GknknQkn)≥ρknwkn,∀n,k, (13b) ρkn≥0,∀n,k, (13c)

Due to the monotonically increasing property of logarithm functions, the value of the objective function (13a) increases with . Based on constraint (13b), the upper bound of is . Therefore, for maximizing the objective function in P2a with (13b) and (13c), it is equivalent to maximize the objective function in P2.

We note that the functions on both sides of constraint (13b) and on the left-hand side of constraints (11b) and (11c) are bilinear functions with respect to , , and . Therefore, in the following subsection, we exploit the bilinearity of these optimization variables to design a tractable resource allocation.

### Iv-B Successive Convex Approximation

Recall that the beamforming matrix and power allocation coefficient are coupled together as bilinear functions in constraints (11b), (11c), and (13b), e.g. and . In fact, the Hessian matrix of a bilinear function is neither a positive nor a negative semidefinite matrix. Thus, bilinear functions are neither convex nor concave in general, which is an obstacle in designing a computationally efficient resource allocation algorithm.

Now, we handle the bilinear terms in the following. We note that the bilinear function on the left-hand side of (13b) is desired to be transformed into a concave function. Whereas and on the left-hand side of (11b) and (11c), respectively, are desired to be transformed into convex functions. In order to convexify the considered constraints, we first adopt the Schur complement [40] to handle the bilinear constraint in (13b) which leads to the following equivalent constraints:

 [akntkntknTr(GknknQkn)]⪰0, ∀n,k, (14) t2knwkn≥ρkn, ∀n,k, (15)

where is an auxiliary variable.

However, (15) is still a non-convex constraint since it is a difference-of-convex functions (DC) [41]. To address this issue, we then tackle it via the SCA method based on the first-order Taylor expansion [42]. In particular, on the left-hand side of (15) is convex in both and , and thus can be tightly bounded from below with its first-order approximation. Specifically, for any fixed point with and , we have

 t2knwkn≥2~tkn~wkntkn−~t2kn~w2knwkn≥ρkn. (16)

By applying the concept of SCA [39, 42], we iteratively update the fixed points and in the -th iteration as

 ~w(m)kn=w(m−1)kn,  ~t(m)kn=t(m−1)kn. (17)

For handling the bilinear functions on the left-hand side of (11b) and (11c), we adopt the SCA approach based on arithmetic-geometric mean (AGM) inequality such that the original non-convex feasible set is sequentially upper bounded by a convex set. To this end, the non-convex bilinear functions in (11b) and (11c) are replaced by their corresponding convex upper bounds which are given by

 2aknTr(GknknQkn) ≤(aknckn)2+(∗Tr(GknknQkn)ckn)2, (18)
 2aknTr(HknknQkn) ≤(akndkn)2+(∗Tr(HknknQkn)dkn)2, (19)

where and , , , are fixed feasible points. To tighten the upper bounds, we iteratively update the fixed feasible points and . The update equations in the -th iteration are given by

 c(m)kn =  ⎷Tr(GknknQ(m−1)kn)a(m−1)kn, d(m)kn =  ⎷Tr(HknknQ(m−1)kn)a(m−1)kn, (20)

where the derivations are given in Appendix A. Then, the new constraints in the -th iteration are given by

 (aknc(m)kn)2+⎛⎜⎝Tr(GknknQkn)c(m)kn⎞⎟⎠2≤ 2Tr(GknknQkn)1+γ−2γ1+γwkn,∀n,k, (21) (aknd(m)kn)2+⎛⎜⎝Tr(HknknQkn)d(m)kn⎞⎟⎠2≤ 2Tr(HknknQkn)1+γ−2γ1+γrkn,∀n,k, (22)

Based on the aforementioned transformations and approximations, constraints given in (11b), (11c), and (13b) can be approximated by some convex constraints. Now, the final difficulty to proceed arises from the rank-one constraint in (11g), which is combinatorial. To address this issue, we drop the constraint to obtain a relaxed version of P2a in (23), which is denoted by P3 and given at the top of next page.

Now, the optimization problem P3 is convex for any given , , , and , and thus can be solved efficiently by off-the-shelf solvers for solving convex programs, e.g. CVX [43].

We note that the rank constraint is dropped in P3 and the obtained solution may not satisfy the rank constraint. We next prove that the solution obtained in P3 can always satisfy the dropped rank constraint.

###### Theorem 1

The optimal solution obtained in P3 is always a rank-one matrix, despite the relaxation of the rank constraint.

###### Proof:

Please refer to Appendix B. \qed

Then, we employ an iterative algorithm to tighten the obtained upper bounds, i.e., (21), (22), (16), as summarized in Algorithm 1. In each iteration, the proposed iterative scheme generates a sequence of feasible solutions to the convex optimization problem P3 successively.

### Iv-C Algorithm Convergence Analysis

In the above sections, we tackle the optimization problem P1 via transforming it into P2 and then approximate it by P3. We now discuss the connections among these optimization problems in the following lemma.

###### Lemma 1

Algorithm 1 converges to a stationary point satisfying the Karush-Kuhn-Tucker (KKT) conditions of P1.

###### Proof:

Please refer to Appendix C. \qed

In other words, Algorithm 1 is able to achieve a suboptimal solution of P1 with polynomial-time computational complexity.

## V Numerical Results

In this section, we numerically examine the performance of our designed transmission scheme. In the simulation, we consider both small scale fading (i.e., Rayleigh fading) and path loss in the channels. We model the channels as and , where and represent the distances from BS to and , respectively, and represent the Rayleigh fading coefficients from BS to and , respectively, and is the path loss exponent. Here, the entries in and are modeled as CSCG random variables with zero mean and unit variance. We set the distance between every two neighboring BSs as m. We assume that the locations of users in each cell are randomly and uniformly distributed in discs with radius m centered at the location of the BS. The iteration error tolerance, i.e., , in Algorithm 1 is . Without loss of generality, we also assume that the noise powers at all users are the same with and the maximum transmit powers at all BSs are identical with . For the sake of presentation, we define the average transmit SNR as and denote the proposed joint beamforming and power allocation design scheme as “NOMA-CoMP”. Besides, we set the sum-rate of users in Group 1 to zero if the optimization problem in (23) is infeasible to account the penalty of failure.

### V-a Convergence

We evaluate the convergence rate of the proposed NOMA-CoMP. In Fig. 2, we plot the sum-rate of users in Group 1 versus the iteration index for different values of and . We observe that the convergence rate is faster when , , compared to , and