Stochastic Routing and Scheduling Policies for Energy Harvesting Communication Networks

# Stochastic Routing and Scheduling Policies for Energy Harvesting Communication Networks

## Abstract

In this paper, we study the joint routing-scheduling problem in energy harvesting communication networks. Our policies, which are based on stochastic subgradient methods on the dual domain, act as an energy harvesting variant of the stochastic family of backpresure algorithms. Specifically, we propose two policies: (i) the Stochastic Backpressure with Energy Harvesting (SBP-EH), in which a node’s routing-scheduling decisions are determined by the difference between the Lagrange multipliers associated to their queue stability constraints and their neighbors’; and (ii) the Stochastic Soft Backpressure with Energy Harvesting (SSBP-EH), an improved algorithm where the routing-scheduling decision is of a probabilistic nature. For both policies, we show that given sustainable data and energy arrival rates, the stability of the data queues over all network nodes is guaranteed. Numerical results corroborate the stability guarantees and illustrate the minimal gap in performance that our policies offer with respect to classical ones which work with an unlimited energy supply.

## 1Introduction

Providing wireless devices with Energy Harvesting (EH) capabilities enables them to acquire energy from their surroundings. The sources from which to obtain such energy can be of a varied nature, with some of the most common being thermal, vibrational or solar sources [2]. Such ample variety of energy sources, coupled with recent hardware advancements, enables devices to acquire sufficient energy to power themselves. This, in turn, frees these devices from the constraints that traditional battery-only operation imposes. Nonetheless, the random and intermittent nature of this new energy supply calls for a new approach to the design of communication policies.

As a consequence, there is significant interest in the study of communication devices powered by energy harvesting. The scenarios of EH-aware communication studied in the literature are vast and range from throughput maximization [3], source-channel coding [8], estimation [12], simultaneous information and power transfer [15] and many others (see [19] for an overall review of current research efforts).

The appearance of multiple interconnected devices powered by energy harvesting results in communication networks formed by self-sustainable and perpetually communicating nodes. In such scenarios, there is the necessity of designing efficient routing and scheduling algorithms that explicitly take into account the energy harvesting process. In this sense, there have been some previous efforts in developing communication policies for these types of multi-hop networks. In general, the full characterization of the optimal transmission policies is a difficult problem, as optimal transmission policies are heavily coupled throughout the network. Under full non-causal knowledge of the energy harvesting process, the optimal transmission policies of a simpler two-hop network have been studied in [20]. A more realistic approach is the consideration of non-causal knowledge of the energy harvesting process. Under this assumption, the authors in[21] jointly optimize data compression and transmission tasks to obtain a close-to-optimal policy. In [22], the authors propose an EH-aware routing scheme that is asymptotically optimal with respect to the network size. The authors in [23], address the EH scheduling problem for both single-hop and multi-hop networks, and provide a joint admission control and routing policy. Also, in the same line, the authors in [24] propose a policy which improves on the multi-hop performance bounds of [23]. Overall, non-causal policies are typically designed under the assumption of independent and identically distributed (i.i.d.) or Markov energy harvesting and data arrival processes, and Lyapunov optimization techniques are used to derive their queue stability results.

In this paper, we study the problem of jointly routing and scheduling data packets in an energy harvesting communication network. We start by introducing the system model in Section 2. We consider a communication network where each node independently generates traffic for delivery to a specific destination and collaborates with the other nodes in the network to ensure the delivery of all data packets. In this way, each node decides the next suitable hop for each packet in its queue (routing), and when to transmit it (scheduling). The solution to this problemwhen the nodes are not EH-poweredis given by the backpressure (BP) algorithm [25]. When the nodes are powered by energy harvesting, the previous works [23] and [24] considered a similar problem, which consists in finding admission control and resource allocation policies that satisfy network stability and energy causality while attaining close-to-optimal performance. In our work, instead, the goal is to find stabilizing policies given the data rates. Also, while previous works [23] require data and energy arrival processes to be i.i.d. or Markov, we only require them to be ergodic, which is a weaker requirement. Furthermore, our approach to the problem is also markedly different. While the works [23] and [24] relied on queueing theory and Lyapunov drift arguments to find stabilizing policies, we instead interpret the scheduling and routing problem as a stochastic optimization problem. This allows us to resort to a dual stochastic subgradient descent algorithm [26] to solve the joint routing-scheduling problem.

We devote Section 3 to the development of the proposed stochastic joint routing and scheduling algorithms. The main issue to tackle is that the introduction of energy harvesting constraints results in a causality problem regarding the energy consumption. In order to solve this, we introduce a modified problem formulation that allows us to ensure causality. Under this framework, we propose two different policies. The first, which we denote Stochastic Backpressure with Energy Harvesting (SBP-EH), is a policy of rather simple nature. The network nodes track the pressure of the data flows by computing the difference between the Lagrange multipliers associated to their queue stability constraints and the ones of their neighbors (instead of their data queues as in the classical backpressure algorithm). Then, the Lagrange multipliers associated with the battery state reduce the pressure when the stored energy in the node decreases. The resulting routing-scheduling decision is to transmit the flow with highest pressure. The second policy, which we name Stochastic Soft Backpressure with Energy Harvesting (SSBP-EH), is a probabilistic policy. In this policy, the nodes perform the same tracking of pressure as the SBP-EH policy. However, instead of transmitting the flow with the highest pressure, the flows are equalized in an inverse waterfilling manner. This results in a routing-scheduling probability mass function, where the transmit decision is taken as a sample of this distribution. This second policy, while not as simple as the previous one, provides several improvements in the stabilization speed of the network, as well as a reduction in the packets in queue and packet delivery delay once the network is stabilized.

Theoretical guarantees, namely, queue stability and energy causality are discussed in Section 4. For both policies, we provide the necessary battery capacity which ensures the proper behavior of the algorithms. Furthermore, we also certify that given sustainable data and energy arrival rates, the stability of the data queues over all network nodes is guaranteed. After this, we dedicate Section 5 to simulations assessing the performance of our proposed policies and verify that they show a minimal gap in performance with respect to classical policies operating with an unlimited energy supply. Finally, we provide some concluding remarks in Section 6.

## 2System Model

Consider a communication network given by the graph , where is the set of nodes in the network and is the set of communication links, such that if node is capable of communicating with node , we have . Moreover, we define the neighborhood of node as the set . The network supports information flows (which we index by the set ), where for a flow , the destination node is denoted by . At a time slot , each flow at the -th node generates packets to be delivered to the node . This packet arrival process is assumed to be stationary with mean . At the same time, the -th node routes packets to its neighbors , while simultaneously being routed packets. For simplicity, at each time slot, we restrict each node to route one single packet to its neighbors. Therefore, the nodes have the following routing constraint

Furthermore, each node in the network keeps track of the number of packets awaiting to be transmitted for each flow. Denoting by the -th flow data queue at the -th node and time slot , the evolution of the queue is given by

for all and . The objective is to determine routing policies such that the queues in remain stable while satisfying the routing constraints given by . By grouping all the queues in a vector , we say that the routing policies guarantee stability if there exists a constant such that for some arbitrary time we have

This is to say that, almost surely, no queue becomes arbitrarily large. In turn, we can guarantee this if the average rate at which packets enter the queues is smaller than the rate at which they exit them. In order to formally state this, let us denote the ergodic limits of processes and by

Then, in order to have stable data queues in the network, it suffices to satisfy the condition

for all and . If there exist routing variables satisfying this inequality, then the queue evolution in follows a supermartingale expression, and the stability condition given by is then guaranteed by the martingale convergence theorem [27]. Alternatively, by introducing arbitrary concave functions , we can formulate this as the following optimization problem

Observe that in the inequality allows for equality whereas in the inequality is strict. This mismatch is necessary because optimization problems are not well behaved on open sets. We can then think of as a relaxation of but one of little practical consequence as it is always possible to add a small slack term to to produce a non-strict inequality that implies the strict inequality in . We don’t do that to avoid a cumbersome term of little conceptual value. We emphasize that implicit to is the constraint for all , which is the same as but for average variables. We will ensure later that the algorithm we design satisfies the constraint not just on average but for all time instances – see Section 3.

Assuming data arrival rates satisfying for all as well as the inequalities in exist, the objective is to design an algorithm such that the instantaneous routing variables satisfy and the routing constraints in are satisfied for all time slots. This is the optimization problem that the backpressure family of algorithms solve. By resorting to a stochastic subgradient method on the dual domain, a direct comparison can be established between data queues and Lagrange multipliers [28]. Then, the choice of objective function in the optimization problem determines the resulting variant of the backpressure algorithm. For example, on one hand, the stochastic backpressure (SBP) algorithm [25] can be recovered by the use of a linear objective function. On the other hand, the choice of a strongly concave objective function leads to the soft stochastic backpressure (SSBP) algorithm [29].

### 2.1Routing and Scheduling with Energy Harvesting

Different from classical approaches [25], we consider that the network nodes are powered by energy harvesting. At time slot , the -th node harvests units of energy, where the energy harvesting process is assumed to be stationary with mean . We consider a normalized energy harvesting process, where the routing of one packet consumes one unit of energy. Furthermore, we consider packet transmission to be the only energy-consuming action taken by the nodes. Under these conditions and denoting by the energy stored in the -th node’s battery at time , the following energy causality constraint must be satisfied for all time slots

Additionally, we consider that nodes have a finite battery of capacity . Then, we can write the battery dynamics as

for , where denotes the projection to the interval . In order to introduce these constraints into the optimization problem , we denote the ergodic limit of the energy harvesting process by

Then, substituting the battery dynamics given by in the energy causality constraint and then taking the ergodic limits on both sides of the inequality, we obtain the following average constraint in the routing variables

This states that the average amount of energy spent must be less than the average energy harvested. Then, we introduce this constraint into problem , resulting in the following optimization problem

Assuming data and energy arrival rates satisfying and exist, the goal is to design an algorithm such that the instantaneous routing variables satisfy and the constraints and are satisfied for all time slots. However, the use of the average energy constraint presents a causality problem, as a solution satisfying does not guarantee that the energy causality constraint in is satisfied for all time slots. In order to circumvent this, we propose the introduction of the following modified optimization problem

This optimization problem differs from in the introduction of an auxiliary variable . This variable is restricted to lie in the interval , with being a constant whose value is determined by the system parameters. This auxiliary variable appears in the queue stability constraint , where it helps to satisfy the constraint if necessary. Furthermore, we have added the term in the objective function, where is a constant parameter. The value of this parameter is chosen such that the optimal value of the Lagrange multipliers of the queue constraint lies in the interval . We later show in Section 4 that this allows us to satisfy the energy causality constraints while also stabilizing the data queues.

## 3Joint Routing and Scheduling Algorithm

As we mentioned previously, in order to solve optimization problem posed in we resort to a primal-dual method. To start, let us define the vector collecting the routing variables and auxiliary variables and the vector collecting the queue multipliers associated with constraint and battery multipliers corresponding to constraint . Furthermore, we collect the implicit optimization constraints in the set . Then, we write the Lagrangian of the optimization problem as follows

The Lagrange dual function is then given by

An immediate issue that arises when trying to solve this problem is that network nodes have no knowledge of the data arrival rates nor the energy harvesting rates . Nonetheless, the nodes observe the instantaneous rates and , hence we resort to using these instantaneous variables. Furthermore, we can reorder the Lagrangian to allow for a separate maximization over network nodes, where each node only needs the queue multipliers of its neighboring nodes. The routing variables can then be obtained as follows

for . In a similar way, the auxiliary variables at each node are given by

This is simply a threshold operation, where if and if . Now, since the dual function in is convex, we can minimize it by performing a stochastic subgradient descent. Then, the dual updates are given by the following expressions

where is the projection on the nonnegative orthant. For compactness, we also express the dual updates in vector form as , where corresponds to the vector collecting the stochastic subgradients. Since the algorithm that we propose is designed to be run in an online fashion, we have considered a fixed step size in the dual updates. Specifically, we have used a unit step size. This allows a clear comparison between dual variables and data queues and battery dynamics as outlined in Figure ?. For the case of the data queues, the difference between their dynamics and those of their Lagrange multiplier counterparts is given by the auxiliary variable in the dual update. Assume a packet is either routed or not, i.e., . Then the dual variables follow the data queues until , at which point, the dual variables are pushed back by the auxiliary variable . From this point forward, the queue and multiplier dynamics lose their symmetry, coupling again when the queue empties. In a similar way, a comparison can also be drawn between the battery dynamics and the battery dual update . In this case, the symmetry exists in a mirrored way, as the relationship between the battery state and its multipliers is given by . Different from the case of data queues, the coupling between the battery state and its multipliers is never lost.

Next, we consider some choices of the objective function in the optimization problem which lead to familiar formulations of the backpressure algorithm adapted to the energy harvesting process. The steps of the two resulting policies are summarized in Algorithm ?.

### 3.1Stochastic Backpressure with Energy Harvesting (SBP-EH)

Consider functions which are linear with respect to the routing variables, i.e., taking the form , where is an arbitrary weight. In this case, we recover a version of the stochastic backpressure algorithm adapted to the energy harvesting process. For a linear objective function, the maximization in leads to the routing variables

To solve the maximization in it suffices to find the flow over the neighboring nodes with the largest differential and if it is positive, set its corresponding routing variable to one while the other variables are kept to zero. This algorithm, when , is analogous to the stochastic form of backpressure. In the classical backpressure algorithm, the flow with the largest queue differential is chosen. Interpreted in its stochastic form, the flow with the largest Lagrange multiplier difference is chosen. In the SBP-EH policy, the stochastic form of backpressure adds the battery multiplier . As the battery depletes, the value of increases and the pressure to transmit of this node decreases.

### 3.2Stochastic Soft Backpressure with Energy Harvesting (SSBP-EH)

Now, we consider a quadratic plus linear term function given by . This leads to a stochastic soft backpressure algorithm [29], where the routing variables obtained by the maximization in are given by

where are the Lagrange multipliers ensuring for all . This expression can be understood a form of inverse waterfilling. An example of this solution is shown in Figure 1. Let us construct rectangles of height and scale them by the widths . For each node, every possible flow and neighbor routing destination is represented by one of these rectangles. Then, water is poured from the bottom, in an inverse manner until the waterlevel is reached. The resulting area of water filled inside the rectangles represents the probability mass function of the routing variables. Then, the node takes its routing decision by drawing a sample from this distribution.

While not as simple as the SBP-EH algorithm, the SSBP-EH algorithm presents an important improvement over the former. The introduction of an strongly concave objective function allows the dual function in to be differentiable. This, in turn, makes the algorithm take the form of an stochastic gradient rather than a stochastic subgradient (which is the case of SBP-EH), therefore improving the expected rate of stabilization of the algorithm from to [30].

## 4Causality and Stability Analysis

In this section, we provide theoretical guarantees on the behavior of the proposed policies. On one hand, we establish the conditions under which the routing policies generated by Algorithm ? satisfy the energy causality constraints . And, on the other hand, we provide stability guarantees on the network queues.

### 4.1Energy Causality

As we mentioned previously in Section 2, the presence of the energy harvesting constraints in the stochastic optimization problem introduces the question of causality. In order to have a tractable problem, we have introduced the energy harvesting constraints in an average sense to the routing-scheduling problem. This includes an additional issue, as not all possible solutions satisfy the original causality constraints for all time slots. In order to deal with this, we have modified the problem formulation with the introduction of an auxiliary variable. By appropriately choosing the domain of this auxiliary variable and the nodes’ battery capacity, we can ensure that the causality constraints are satisfied.

To satisfy the energy causality constraints it suffices to show that no transmission occurs when there is no available energy in the battery. This is to say that for all if . In expressions and , corresponding to the SBP-EH and SSBP-EH algorithms, it suffices to ensure that when the battery is empty. In this case, when , the battery dual update takes the value . By the dual update and the minimum value of , the data arrival bound and the number of neighbors , we can upper bound the multiplier difference by over all time slots . We can write then , and since , this ensures that . Hence, satisfying the energy causality constraint for all time slots.

In order to ensure that the energy causality constraints are satisfied, the stochastic subgradients are required to be bounded. This, in turn, forces the probability distribution of the data arrival process to be bounded above by a constant . In practice, for the case in which the probability distribution is not bounded, when a time slot with over packets occurs, only data packets can be kept in the queue and the rest must be discarded to satisfy the energy causality constraints.

### 4.2Queue Stability

Now, we provide guarantees on the queue stability of the proposed policies. Different from other works (such as[23]), which analyze queue stability with Lyapunov drift notions, we resort to duality theory arguments. We do this by leveraging on the fact that the proposed algorithm is a type of stochastic subgradient algorithm. The approach we take to showing that our algorithm makes the queues stable in the sense of is to show that the solution provided by Algorithm ? satisfies the queue stability constraints almost surely. Then, we show that if the optimal queue multipliers are upper bounded by , the solution provided by Algorithm ? also satisfies the stability constraint without auxiliary variable . Hence, the data queues satisfy the stability condition .

First, we start by recalling a common property of the stochastic subgradient.

Take the Lagrangian and substitute the ergodic definitions and . Then, the resulting Lagrangian is given by

Now, recall that the dual function is then given by , and consider the dual function at time , given by . The primal maximization of this dual function is given by the variables and in and , respectively. Hence, we can write the dual function as

where we have moved the expectation operator out of the subgradients due to its linearity. Then we can use the compact notation for the multiplier vector and the subgradient , and substitute the conditional expected value of the subgradients to obtain

For any arbitrary we simply have

Then it simply suffices to subtract expression from to obtain inequality .

Proposition ? shows that the stochastic subgradient is an average descent direction of the dual function . Now, we proceed to quantify the average descent distance of the dual update.

Start by considering the squared distance between the dual variables at time and their optimal value. This distance is given by . Then, we substitute the dual variable by its update . Then, since the projection is nonexpansive we can upper bound the aforementioned distance by

Then, we simply expand the square norm to obtain the expression

Now, by taking the expectation conditioned by on both sides we obtain

And then by substituting the second term on the right hand side by the bound and the third term by the application of Proposition ? with , we have expression .

Then, we leverage on this lemma to show that Algorithm ? converges to a neighborhood of the optimal solution of the dual function.

For ease of exposition, let . Then, define the stopped process , tracking the distance between the dual variables at time and their optimal value, i.e., , until the optimality gap falls below . This expression is given by

where denotes the indicator function. In a similar way, define the sequence which follows until the optimality gap becomes smaller than ,

Now, let be the filtration measuring and . Since and are completely determined by , and is a Markov process, conditioning on is equivalent to conditioning on . Hence, by application of Lemma ?, we can write . Since by definitions and , the processes and are nonnegative, the sequence follows a supermartingale expression. Then, by the supermartingale convergence theorem [27], the sequence converges almost surely, and the sum is almost surely finite. The latter implies that almost surely. Given the definition of , this is implied by either of two events. (i) If the indicator function goes to zero, i.e., for a large ; or (ii) . From any of those events, expression follows.

The convergence of the dual function as asserted in the previous lemma allows us to prove that the sequences of routing decision and auxiliary variables generated by Algorithm ? are almost surely feasible.

First, let us collect the feasible routing variables and auxiliary variables in the vector . Then, if there exist strictly feasible variables , we can bound the value of the dual function as follows. The dual function is defined as the maximum over primal variables , hence . From this, by using the and terms we establish the following bound

Then, by simply reordering terms we obtain the following upper bound on the dual variables

Lemma ? certifies the existence of a time for which . Hence,

for . Now, recall that the feasibility conditions and are given by the limits

which, by recalling that the constraints are simply the stochastic subgradients of the problem, they can also be written in compact form as . Now, consider the dual updates and given by . Since the operator corresponds to a nonnegative projection, the dual variables can be lower bounded by removing the projection and recursively substituting the updates

To prove almost sure feasibility, we will follow by contradiction. First, assume that conditions and are infeasible. In compact form, this means the existence of a time , for which there is a constant such that . By substituting in , we have that the dual variables are lower bounded by . Now, we can freely choose a time such that

for all . However, this contradicts the upper bound established in . This means that there do not exist sequences generated by Algorithm ? such that and are not satisfied. Therefore, the constraints and are satisfied almost surely.

Finally, it suffices to show that if the optimal dual variables are upper bounded by the constants , the system satisfies the original problem without the auxiliary variable. Thus, satisfying the original constraint and hence the queue stability condition .

Take the difference between the Lagrangian of the optimization problem with the auxiliary variable and the original problem . The difference between them is given by

where and are the Lagrange multipliers of the and constraints, respectively. In order for both problems to be equivalent, the minimization of , which is the solution of the dual problem, must be zero. This implies the existence of Lagrange multipliers satisfying the constraints , for all and . Since , the constraints can be satisfied by letting , and acting as a slack variable. Then, , which implies that the optimal solution of both problems is the same. Since [26] and , by Proposition ? the routing variables of Algorithm ? satisfy the constraint .

Denote by the filtration measuring . Then, since the routing variables generated by Algorithm ? satisfy , the queue evolution obeys the supermartingale expression . By the supermartingale convergence theorem [27], the sequence converges almost surely, therefore satisfying the stability condition .

Given an appropriate choice of and feasible data and energy arrivals, Proposition ? guarantees that the nodes route in average as many packets as they receive from neighbors and the arrival process (i.e., the constraint is satisfied). Then, Corollary ? shows that this implies that the queues themselves are almost surely stable.

## 5Numerical Results

In this section, we conduct numerical experiments aimed at evaluating the performance of the proposed SBP-EH and SSBP-EH policies. As a means of comparison, when indicated, we also provide the non-energy harvesting counterparts of our proposed policies. Namely, the Stochastic Backpressure (SBP) [25] and Stochastic Soft Backpressure (SSBP) [29] policies. These policies correspond to solving , the original optimization problem without the energy harvesting constraints, with the objective functions shown in Sections Section 3.1 and Section 3.2, respectively. Hence, these policies assume the availability of an unlimited energy supply. We consider the communication network shown in Figure 2, where we let nodes 1 and 14 act as sink nodes and the rest of the nodes support a single flow with packet arrival rates of packets per time slot. Moreover, we consider the nodes to be harvesting energy at a rate of units of energy per time slot and storing it in a battery of capacity . Furthermore, we set the routing weights to , and let .

### 5.1Network Queues

First, we plot in Figure 3 a sample path of the total number of queued packets in the network as a function of the elapsed time. As expected, all the policies are capable of stabilizing the queues in the network. Due to the random nature of the processes, it is difficult to say exactly at which point stabilization occurs. Nonetheless, for the SBP and SBP-EH policies, the data queues seem to stop growing after around time slots. In the case of the SSBP and SSBP-EH policies, stabilization occurs much more rapidly rapidly, with less than time slots necessary to obtain stability. Also, both soft policies (SSBP and SSBP-EH) stabilize the queues with a lower number of average queued packets than their counterpart non-soft policies (SBP and SBP-EH). Namely, at , the average queued packets are for SBP and for SBP-EH. In the case of the soft policies, these numbers are much smaller, with and packets for SSBP and SSBP-EH, respectively. This also shows that the gap between the SSBP and SSBP-EH policies seems to vanish asymptotically ( at ), while this is not the case for the non-soft policies (a gap of at ). This occurs due to the fact that the SBP and SBP-EH policies choose their routing policy by maximizing the difference between queue multipliers. Hence, making the decision indifferent to the actual value of the multipliers as long as their differences stay the same. For the SSBP and SSBP-EH policies, this situation does not occur due to their randomized nature. Hence, pushing for lower average queued packets. Furthermore, since the data arrivals can be sustained by the energy harvesting process, the SSBP-EH policy tries to get as close a the non-EH one, leading to the small of the gap. Also, note that the SBP-EH and SSBP-EH policies are more volatile than their non-EH counterparts. For example, around , the number of queued packets spikes for the energy harvesting policies, which is not the case in the non-EH ones. These types of spikes arise due to a certain lack of energy around those time instants.

In Figure ? we have plotted the average queued packets at each node for the SBP-EH and SSBP-EH policies. In general, SSBP-EH shows a lower number of average queued packets over all the nodes and the improvements are more significant the lower the pressure the node supports. This tends to translate to better improvements for nodes far away from a sink that tend to be routed less traffic. For example, the nodes and (See Figure 2), which are the furthest away from any sink, show a reduction of and average packets, respectively, when using SSBP-EH. The rest of the nodes also show significant improvements when using SSBP-EH. Nodes , , and , all lying at two hops of distance of a sink are more critical for accessing a sink, as having them congested blocks the access to the sink of the previous nodes and . In this case, the improvements range from to average data packets. Finally, there are the nodes that lie at one hop distance from any sink (nodes , , , , and ) . These nodes sustain a significant amount of traffic and show improvements ranging from to . With the nodes with the highest traffic, nodes and , improving by and data packets, respectively.

The differences between SBP-EH and SSBP-EH are also evidenced in terms of their energy use. In Figure 4 we plot the total energy in the network at a given time slot for both the SBP-EH and the SSBP-EH policies. On one hand, this figure illustrates the high variability in the energy supply due to the energy harvesting process. On the other hand, the SSBP-EH policy is shown to be more aggressive in its energy use. Also, note that drops in total network energy are not necessarily correlated with increases in queued packets in the network. For example, the previously noticed peak of queued data packets at in Figure 3 does not have an equivalent large drop in network energy. This is due to the fact that it is better for energy in the network to overall be lower than to have a specific high-pressure node have an energy shortage. In general, spikes in queued data packets tend to occur when a specific route becomes blocked by the temporary lack of energy.

### 5.2Network Balance

As discussed in Section 4, the choice of the parameters , which control the maximum values taken by the queue multipliers , is important to ensure the stability of the data queues. Namely, the optimal multipliers must be smaller than this parameter. In Figure 5, we plot the multipliers for one of the nodes which supports the most traffic in the network (node 5). The time-average of these dual variables converges to the optimal value. In the chosen scenario, the parameter used, , is well above the optimal value. Hence, the system satisfies Proposition ?, and can be ensured to stabilize the queues. Some additional insight into the importance of the queue multipliers can be gained by a pricing interpretation of the dual problem. Under this interpretation, the dual variables represent the unit price associated to the routing constraint . When the node does not satisfy this constraint, it pays per unit of constraint violation. Likewise, if it strictly satisfies this constraint, it receives per unit of constraint satisfaction. In this sense, the parameter represents both the maximum payment that a node can receive and the maximum price it can pay. Hence, the optimal value of must necessarily fall below in order to obtain a stable system. We can use this pricing interpretation to compare the different policies. In general, the energy harvesting policies have higher values than their non-EH counterparts. This is due to the fact that, due to the energy harvesting constraints, the unit violation of the routing constraint is harder to recoup in the EH-aware policies, hence the higher price paid. In a similar note, due to their more aggressive routing decisions, the soft policies also show higher values than their non-soft counterparts.

Also of interest is the study of the balance characteristics of the network. As discussed previously, the stability guarantees of the network are subject to the existence of a feasible routing solution given the data and energy arrival rates. This motivates another way of showing stability, different from the data queues shown in Figure 3. We can consider that a successful routing strategy is expected to route to the sink nodes as many packets as generated by the network. This is given by the network balance expression , where . The time average of this measure is shown in Figure 6. As expected, the time average data network balance goes to zero for all policies. This illustrates that all policies are capable of routing to the sink nodes as many packets as they arrive to the network, hence ensuring queue stability. We previously observed in Figure 6 that stability occurs around time slots for the SBP and SBP-EH policies and less than time slots for the SSBP and SSBP-EH ones. Those observations can be compared with the network balance of Figure 6, where those values correspond to the time around when the slope of the data balance curve starts to go flat. Remarkably, the proposed energy harvesting policies do not lose convergence speed when compared to the non-EH ones. Also, convergence of the SSBP and SSBP-EH policies occurs at a faster rate, a point that we previously raised in Section 3.2.

Another measure of network balance of interest is related to the energy balance in the network. This can be expressed by . This measure serves to quantify how much of the energy harvested in the network is actually being used. The time average of the energy balance is shown in Figure 7. As expected, given that the network harvests enough energy to support the routing-scheduling decisions, both policies converge to a non-zero value. Once stabilized, the SBP-EH policy has, in average, energy left for around 12 packet transmission in all of the network, while the SSBP-EH only has energy left for an average of 2 packet transmissions. We previously identified in Figure 4 the SSBP-EH to be more aggressive in its energy use. At the same time, we can also say that the SSBP-EH policy uses its energy supply in a more efficient manner. Since the nodes are powered by energy harvesting instead of a limited energy supply, not using available energy can be considered wasteful, as batteries will tend to overflow. In this sense, to use more energy (as in SSBP-EH) rather than to use energy more conservatively (as in SBP-EH), can be seen as a better option. In this sense, SSBP-EH makes a more efficient use of the available energy, resulting in an overall better performance.

### 5.3Network Delay

An additional important characteristic of routing-scheduling policies is their resulting delay in the packet delivery. While the average delay is proportional to the average number of queued packets in the network, we also study this measure explicitly. In order to do this, and under the assumption of first-in first-out queues, we compute the number of time slots it takes for a packet to be delivered to a sink node. We plot in Figure 8 the resulting histogram. In average, the number of time slots it takes to deliver a packet to a sink node is for the SSBP-EH policy, while it is for the SBP-EH policy. This is about a time slot of difference between the policies. Taking a more detailed look at the histogram, we can see that the distribution for the SSBP-EH is very similar to the one of the SBP-EH, but with a time slot shift to the left. As already seen in Fig. ?, the more aggressive behavior of the SSBP-EH policy leads to an overall reduction in the network queues. These smaller queues result in a reduction of the waiting time of packets at each hop, which results in a smaller delivery delay.

## 6Conclusions

In this work, we have generalized the stochastic family of backpressure policies to energy harvesting networks. Different from other works, which are based on Lyapunov drift notions, we have resorted to duality theory. This has allowed us to study the problem under a framework based on the correspondence between queues and Lagrange multipliers. Under this framework, we have proposed two policies, (i) SBP-EH, an easy to implement policy where nodes track the difference between their queue multipliers and the ones of their neighbors. The pressure is further reduced by the battery multipliers as the stored energy decreases. Then, the transmit decision is to transmit the flow with the highest pressure. And (ii) SSBP-EH, a probabilistic policy with improved performance and convergence guarantees, where nodes track the pressure in the same way as SBP-EH but perform an equalization in the form of an inverse waterfilling. This results in a probability mass function for the routing-scheduling decision, where a sample of this distribution is then taken to decide the transmission. For both policies, we have studied the conditions under which energy causality and queue stability are guaranteed, which we have also verified by means of simulations. The numerical results show that given feasible data and energy arrivals, both policies are capable of stabilizing the network. Overall, the SSBP-EH policy shows improvements in queued packets, stabilization speed and delay with respect to the SBP-EH policy. Furthermore, when compared to non-EH policies, the SSBP-EH policy shows to have an asymptotically vanishing gap.

### References

1. M. Calvo-Fullana, J. Matamoros, C. Antón-Haro, and A. Ribeiro, “Stochastic backpressure in energy harvesting networks,” in Acoustics, Speech and Signal Processing (ICASSP). 2017 IEEE International Conference on, March 2017.
2. R. J. Vullers, R. Schaijk, H. J. Visser, J. Penders, and C. V. Hoof, “Energy harvesting for autonomous wireless sensor networks,” IEEE Solid-State Circuits Magazine, vol. 2, no. 2, pp. 29–38, 2010.
3. J. Yang and S. Ulukus, “Optimal packet scheduling in an energy harvesting communication system,” IEEE Transactions on Communications, vol. 60, no. 1, pp. 220–230, 2012.
4. K. Tutuncuoglu and A. Yener, “Optimum transmission policies for battery limited energy harvesting nodes,” IEEE Transactions on Wireless Communications, vol. 11, no. 3, pp. 1180–1189, 2012.
5. O. Ozel, K. Tutuncuoglu, J. Yang, S. Ulukus, and A. Yener, “Transmission with energy harvesting nodes in fading wireless channels: Optimal policies,” IEEE Journal on Selected Areas in Communications, vol. 29, no. 8, pp. 1732–1743, 2011.
6. C. K. Ho and R. Zhang, “Optimal energy allocation for wireless communications with energy harvesting constraints,” IEEE Transactions on Signal Processing, vol. 60, no. 9, pp. 4808–4818, 2012.
7. O. Ozel, K. Shahzad, and S. Ulukus, “Optimal energy allocation for energy harvesting transmitters with hybrid energy storage and processing cost,” IEEE Transactions on Signal Processing, vol. 62, no. 12, pp. 3232–3245, 2014.
8. M. Calvo-Fullana, J. Matamoros, and C. Antón-Haro, “Reconstruction of correlated sources with energy harvesting constraints in delay-constrained and delay-tolerant communication scenarios,” IEEE Transactions on Wireless Communications, vol. 16, no. 3, pp. 1974–1986, 2017.
9. P. Castiglione and G. Matz, “Energy-neutral source-channel coding with battery and memory size constraints,” IEEE Transactions on Communications, vol. 62, no. 4, pp. 1373–1381, 2014.
10. O. Orhan, D. Gunduz, and E. Erkip, “Source-channel coding under energy, delay, and buffer constraints,” IEEE Transactions on Wireless Communications, vol. 14, no. 7, pp. 3836–3849, July 2015.
11. P. Castiglione, O. Simeone, E. Erkip, and T. Zemen, “Energy management policies for energy-neutral source-channel coding,” IEEE Transactions on Communications, vol. 60, no. 9, pp. 2668–2678, 2012.
12. G. Yang, V. Y. Tan, C. K. Ho, S. H. Ting, and Y. L. Guan, “Wireless compressive sensing for energy harvesting sensor nodes,” IEEE Transactions on Signal Processing, vol. 61, no. 18, pp. 4491–4505, 2013.
13. S. Knorn, S. Dey, A. Ahlén, and D. E. Quevedo, “Distortion minimization in multi-sensor estimation using energy harvesting and energy sharing.” IEEE Trans. Signal Processing, vol. 63, no. 11, pp. 2848–2863, 2015.
14. M. Calvo-Fullana, J. Matamoros, and C. Antón-Haro, “Sensor selection and power allocation strategies for energy harvesting wireless sensor networks,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 12, pp. 3685–3695, 2016.
15. K. Huang and E. Larsson, “Simultaneous information and power transfer for broadband wireless systems,” IEEE Transactions on Signal Processing, vol. 61, no. 23, pp. 5972–5986, 2013.
16. J. Xu, L. Liu, and R. Zhang, “Multiuser miso beamforming for simultaneous wireless information and power transfer,” IEEE Transactions on Signal Processing, vol. 62, no. 18, pp. 4798–4810, 2014.
17. G. Zheng, Z. K. M. Ho, E. A. Jorswieck, and B. E. Ottersten, “Information and energy cooperation in cognitive radio networks.” IEEE Trans. Signal Processing, vol. 62, no. 9, pp. 2290–2303, 2014.
18. L. Liu, R. Zhang, and K.-C. Chua, “Secrecy wireless information and power transfer with miso beamforming,” IEEE Transactions on Signal Processing, vol. 62, no. 7, pp. 1850–1863, 2014.
19. S. Ulukus, A. Yener, E. Erkip, O. Simeone, M. Zorzi, P. Grover, and K. Huang, “Energy harvesting wireless communications: A review of recent advances,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 3, pp. 360–381, 2015.
20. O. Orhan and E. Erkip, “Energy harvesting two-hop communication networks,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 12, pp. 2658–2670, 2015.
21. C. Tapparello, O. Simeone, and M. Rossi, “Dynamic compression-transmission for energy-harvesting multihop networks with correlated sources,” IEEE/ACM Transactions on Networking (TON), vol. 22, no. 6, pp. 1729–1741, 2014.
22. L. Lin, N. B. Shroff, and R. Srikant, “Asymptotically optimal energy-aware routing for multihop wireless networks with renewable energy sources,” IEEE/ACM Transactions on networking, vol. 15, no. 5, pp. 1021–1034, 2007.
23. M. Gatzianas, L. Georgiadis, and L. Tassiulas, “Control of wireless networks with rechargeable batteries,” IEEE Transactions on Wireless Communications, vol. 9, no. 2, pp. 581–593, 2010.
24. L. Huang and M. J. Neely, “Utility optimal scheduling in energy-harvesting networks,” IEEE/ACM Transactions on Networking (TON), vol. 21, no. 4, pp. 1117–1130, 2013.
25. L. Tassiulas and A. Ephremides, “Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks,” IEEE Transactions on Automatic Control, vol. 37, no. 12, pp. 1936–1948, 1992.
26. A. Ribeiro, “Ergodic stochastic optimization algorithms for wireless communication and networking,” IEEE Transactions on Signal Processing, vol. 58, no. 12, pp. 6369–6386, 2010.
27. R. Durrett, Probability: theory and examples.1em plus 0.5em minus 0.4emCambridge university press, 2010.
28. L. Huang and M. J. Neely, “Delay reduction via lagrange multipliers in stochastic network optimization,” IEEE Transactions on Automatic Control, vol. 56, no. 4, pp. 842–857, 2011.
29. A. Ribeiro, “Stochastic soft backpressure algorithms for routing and scheduling in wireless ad-hoc networks,” in Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2009 3rd IEEE International Workshop on.1em plus 0.5em minus 0.4emIEEE, 2009, pp. 137–140.
30. D. P. Bertsekas, Convex optimization algorithms.1em plus 0.5em minus 0.4emAthena Scientific, 2015.