Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network
Adaptive traffic signal control, which adjusts traffic signal timing according to real-time traffic, has been shown to be an effective method to reduce traffic congestion. Available works on adaptive traffic signal control make responsive traffic signal control decisions based on human-crafted features (e.g. vehicle queue length). However, human-crafted features are abstractions of raw traffic data (e.g., position and speed of vehicles), which ignore some useful traffic information and lead to suboptimal traffic signal controls. In this paper, we propose a deep reinforcement learning algorithm that automatically extracts all useful features (machine-crafted features) from raw real-time traffic data and learns the optimal policy for adaptive traffic signal control. To improve algorithm stability, we adopt experience replay and target network mechanisms. Simulation results show that our algorithm reduces vehicle delay by up to and when compared to another two popular traffic signal control algorithms, longest queue first algorithm and fixed time control algorithm, respectively.
Traffic congestion has led to some serious social problems: long travelling time, fuel consumption, air pollution, etc [1, 2]. Factors responsible for traffic congestion include: proliferation of vehicles, inadequate traffic infrastructure and inefficient traffic signal control. However, we cannot stop people from buying vehicles and building new traffic infrastructure is of high cost. The relatively easy solution is to improve efficiency of traffic signal control. Fixed-time traffic signal control is common in use, where traffic signal timing at an intersection is predetermined and optimized offline based on history traffic data (not real-time traffic demands). However, traffic demands may change from time to time, making predetermined settings of traffic signal timing out of date. Therefore, fixed-time traffic signal control cannot adapt to dynamic and bursty traffic demands, resulting in traffic congestion.
In contrast, adaptive traffic signal control, which adjusts traffic signal timing according to real-time traffic demand, has been shown to be an effective method to reduce traffic congestion [3, 4, 5, 6, 7, 8]. For example, Zaidi et al.  and Gregoire et al.  proposed adaptive traffic signal control algorithms based on back-pressure method, which is similar to pushing water (here vehicles) to flow through a network of pipes (roads) by pressure gradients (the number of queued vehicles) . Authors in [5, 6, 7, 8] proposed to use reinforcement learning method to adaptively control traffic signals, where they modelled the control problem as a Markov decision process . However, all these works make responsive traffic signal control decisions based on human-crafted features, such as vehicle queue length and average vehicle delay. Human-crafted features are abstractions of raw traffic data (e.g., position and speed of vehicles), which ignore some useful traffic information and lead to suboptimal traffic signal controls. For example, vehicle queue length does not consider vehicles that are not in queue but will come soon, which is also useful information for controlling traffic signals; average vehicle delay only reflects history traffic data not real-time traffic demand.
In this paper, instead of using human-crafted features, we propose a deep reinforcement learning algorithm that automatically extracts all features (machine-crafted features) useful for adaptive traffic signal control from raw real-time traffic data and learns the optimal traffic signal control policy. Specifically, we model the control problem as a reinforcement learning problem . Then, we use deep convolutional neural network to extract useful features from raw real-time traffic data (i.e., vehicle position, speed and traffic signal state) and output the optimal traffic signal control decision. A well-known problem with deep reinforcement learning is that the algorithm may be unstable or even diverge in decision making . To improve algorithm stability, we adopt two methods proposed in : experience replay and target network (see details in Section III).
The rest of this paper is organized as follows. In Section II, we introduce intersection model and define reinforcement learning components: intersection state, agent action, reward and agent goal. In Section III, we present details of our proposed deep reinforcement learning algorithm for traffic signal control. In Section IV, we verify our algorithm by simulations and compare its performance to popular traffic signal control algorithms. In Section V, we review related work on adopting deep reinforcement learning for traffic signal control and their limitations and conclude the whole paper in Section VI.
Ii System Model and Problem Formulation
In this section, we first introduce intersection model and then formulate traffic signal control problem as a reinforcement learning problem.
Consider a four-way intersection in Fig.1, where each road consists of four lanes. For each road, the innermost lane (referred to as ) is for vehicles turning left, the middle two lanes ( and ) are for vehicles going straight and the outermost lane () is for vehicles going straight or turning right. Vehicles at this intersection run under control of traffic signals: green lights mean vehicles can go through the intersection, however vehicles at left-turn waiting area should let vehicles going straight pass first; yellow lights mean lights are about to turn red and vehicles should stop if it is safe to do so; red lights mean vehicles must stop. For example, green lights for west-east traffic are turned on in Fig.1.
We formulate traffic signal control problem as a reinforcement learning problem shown in Fig.2 , where an agent interacts with the intersection at discrete time steps, , and the goal of the agent is to reduce vehicle staying time at this intersection in the long run, thus alleviating traffic congestion. Specifically, such an agent first observes intersection state (defined later) at the beginning of time step , then selects and actuates traffic signals . After vehicles move under actuated traffic signals, intersection state changes to a new state . The agent also gets reward (defined later) at the end of time step as a consequence of its decision on selecting traffic signals. Such reward serves as a signal guiding the agent to achieve its goal. In time sequence, the agent interacts with the intersection as . Next, we define intersection state , agent action and reward , respectively.
Intersection State: Intersection information needed by the agent to control traffic signals includes vehicle position, vehicle speed at each road and traffic signal state. To easily represent information of vehicle position and vehicle speed (following methods in ), we divide lane segment of length , starting from stop line, into discrete cells of length for each road as illustrated in Fig. 3. We then collect vehicle position and speed information of road into two matrices: matrix of vehicle position and matrix of vehicle speed . If a vehicle is present at one cell, the corresponding entry of matrix is set to . The vehicle speed, normalized by road speed limit, is recorded at the corresponding entry of matrix . The matrix of vehicle position for all roads of the intersection is then given by
Similarly, the matrix of vehicle speed for all roads of the intersection is given by
To represent the state of selected traffic signals, we use a vector of size since the agent can only choose between two actions: turning on green lights for west-east traffic (i.e., red lights for north-south traffic) or turning on green lights for north-south traffic (i.e., red lights for west-east traffic). When green lights are turned on for west-east traffic, ; when green lights are turned on for north-south traffic, .
In summary, at the beginning of time step , the agent observes intersection state for traffic signal control, where denotes the whole state space.
Agent Action: As shown in Fig.4, after observing intersection state at the beginning of each time step , the agent chooses one action : turning on green lights for west-east traffic () or for north-south traffic (), and then executes the chosen action. Green lights for each action last for fixed time interval of length . When green light interval ends, the current time step ends and new time step begins. The agent then observes new intersection state and chooses the next action (the same action may be chosen consecutively across time steps, e.g., steps and in Fig.4). If the chosen action at time step is the same with previous action , simply keep current traffic signal settings unchanged. If the chosen action is different from previous action , before the selected action is executed, the following transition traffic signals are actuated to clear vehicles going straight and vehicles at left-turn waiting area. First, turn on yellow lights for vehicles going straight. All yellow lights last for fixed time interval of length . Then, turn on green lights of duration for left-turn vehicles. Finally, turn on yellow lights for left-turn vehicles. An example in Fig. 5 shows the traffic signal timing corresponding to the chosen actions in Fig. 4.
Define action policy as rules the agent follows to choose actions after observing intersection state. For example, can be a random policy such that the agent chooses actions with probability .
Reward: To reduce traffic congestion, it is reasonable to reward the agent at each time step for choosing some action if the time of vehicles staying at the intersection decreases. Specifically, the agent observes vehicles staying time twice every time step to determine its change as shown in Fig. 6. The first observation is at the beginning of green light interval at each time step and the second observation is at the end of green light interval at each time step.
Let be the staying time (in seconds) of vehicle from the time the vehicle enters one road of the intersection to the beginning of green light interval at time step (vehicle should still be at the intersection, otherwise ), and be the staying time of vehicle from the time the vehicle enters one road of the intersection to the end of green light interval at time step . Similarly, let be the sum of staying time of all vehicles at the beginning of green light interval at time step , and be the sum of staying time of all vehicles at the end of green light interval at time step . For example, at time step in Fig. 6, is observed at the beginning of time step because green light interval starts at the beginning of time step and is observed at the end of time step . However, at time step , is observed not at the beginning of time step but when transition interval ends and green light interval begins, is observed at the end of time step , i.e., when green light interval ends. At time step , if the staying time decreases, , the agent should be rewarded; if the staying time increases, , the agent should be penalized. Thus, we define the reward for the agent choosing some action at time step as follows
Agent Goal: Recall that the goal of the agent is to reduce vehicle staying time at the intersection in the long run. Suppose the agent observes intersection state at the beginning of time step , then makes action decisions according to some action policy hereafter, and receives a sequence of rewards after time step , , . If the agent aims to reduce vehicle staying time at the intersection for one time step , it is sufficient for the agent to choose one action that maximizes the immediate reward as defined in (3). Since the agent aims to reduce vehicle staying time in the long run, the agent needs to find an action policy that maximizes the following cumulative future reward, namely Q-value,
where the expectation is with respect to action policy , is a discount parameter, , reflecting how much weight the agent puts on future rewards: means the agent is shortsighted, only considering immediate reward and approaching means the agent is more farsighted, considering future rewards more heavily.
More formally, the agent needs to find an action policy such that
Denote the optimal Q-values under action policy by .
Iii Deep Reinforcement Learning Algorithm for Traffic Signal Control
In this section, we introduce deep reinforcement learning algorithm that extracts useful features from raw traffic data and finds the optimal traffic signal control policy , and experience replay and target network mechanisms to improve algorithm stability.
If the agent already knows the optimal Q-values for all state-action pairs , the optimal action policy is simply choosing the action that achieves the optimal value under intersection state . Therefore, the agent needs to find optimal Q-values next. For optimal Q-values , we have the following recursive relationship, known as Bellman optimality equation ,
The intuition is that the optimal cumulative future reward the agent receives is equal to the immediate reward it receives after choosing action at intersection state plus the optimal future reward thereafter. In principle, we can solve to get optimal Q-values if the number of total states is finite and we know all details of the underlying system model, such as transition probabilities of intersection states and corresponding expected reward. However, it is too difficult, if not impossible, to get these information in reality. Complex traffic situations at the intersection constitute enormous intersection states, making it hard to find transition probabilities for those states.
Instead of solving directly, we resort to approximating those optimal Q-values by a parameterized deep neural network (DNN) such that the output of the neural network , where are features/parameters that will be learned from raw traffic data.
DNN Structure: We construct such a DNN network, following the approach in  and , where the network input is the observed intersection state and the output is a vector of estimated Q-values for all actions under observed state . Detailed architecture of the DNN network is given in Fig. 7: (1) position matrix is fed to a stacked sub-network where the first layer convolves matrix with 16 filters of with stride 2 and applies a rectifier nonlinearity activation function (ReLU), the second layer convolves the first layer output with 32 filters of with stride 1 and also applies ReLU; (2) speed matrix is fed to another stacked sub-network which has the same structure with the previous sub-network, however with different parameters; (3) traffic signal state vector is concatenated with the flattened outputs of the two sub-networks, forming the input of the third layer in Fig. 7. The third and fourth layers are fully connected layers of 128 and 64 units, respectively, followed by rectifier nonlinearity activation functions (ReLU). The final output layer is fully connected linear layer outputting a vector of Q-values, where each vector entry corresponds to the estimated Q-value for an action under state .
DNN Training: The whole training algorithm is summarized in Algorithm 1 and illustrated in Fig. 8. Note that time at line simulates the real world time in seconds, time step at line is one period during which agent events occur as shown in Fig. 4. At each time step , the agent records observed interaction experience into a replay memory . The replay memory is of finite capacity and when it is full, the oldest data will be discarded. To learn DNN features/parameters such that outputs best approximate , the agent needs training data: input data set and the corresponding targets . For input data set, can be retrieved from replay memory . However, target is not known. As in , we use its estimate value as the target instead, where is the output of a separate target network with parameters as shown in Fig. 8 (see for how to set ) and the input of the target network is the corresponding from interaction experience . Define if training episode terminates at time step . The target network has the same architecture with the DNN network shown in Fig. 7. Thus, targets .
After collecting training data, the agent learns features/parameters by training the DNN network to minimize the following mean squared error (MSE)
where is the size of input data set . However, if is large, the computational cost for minimizing is high. To reduce computational cost, we adopt the stochastic gradient descent algorithm RMSProp  with minibatch of size . Following this method, when the agent trains the DNN network, it randomly draws samples from the replay memory to form input data and target pairs (referred to as experience replay), and then uses these input data and targets to update DNN parameters/features by RMSProp algorithm.
After updating DNN features/parameters , the agent also needs to update the target network parameters as follows (we call it soft update) 
where is update rate, .
The explanation for why experience replay and target network mechanisms can improve algorithm stability has been given in .
Optimal Action Policy: Ideally, after the agent is trained, it will reach good estimate of the optimal Q-values and learn the optimal action policy accordingly. In reality, however, the agent may not learn good estimate of those optimal Q-values, because the agent has only experienced limited intersection states so far, not the overall state space, thus Q-values for states not experienced may not be well estimated. Moreover the state space itself may be changing continuously, making current estimated Q-values out of date. Therefore, the agent always faces a trade-off problem: whether to exploit already learned Q-values (which may not be accurate or out of date) and select the action with the greatest Q-value; or to explore other possible actions to improve Q-values estimate and finally improve action policy. We adopt a simple yet effective trade-off method, -greedy method. Following this method, the agent selects the action with the current greatest estimated Q-value with probability (exploitation) and randomly selects one action with probability (exploration) at each time step.
Iv Simulation Evaluation
In this section we first verify our deep reinforcement learning algorithm by simulations in terms of vehicle staying time, vehicle delay and algorithm stability, we then compare the vehicle delay of our algorithm to another two popular traffic signal control algorithms.
Iv-a Simulation Settings
To simulate intersection traffic and traffic signal control, we use one popular open source simulator: Simulation of Urban MObility (SUMO) . Detailed simulation settings are as follows.
Intersection: Consider an intersection of four ways, each road with four lanes as shown in Fig. 1. Set road length to be meters, road segment to be meters, cell length to be meters, road speed limit to be m/s (i.e., km/h), vehicle length to be meters, minimum gap between vehicles to be meters.
Traffic Route: All possible traffic routes at the intersection are summarized in Table I.
|going straight from road to road|
|turning left from road to road|
|going straight from road to road|
|turning left from road to road|
|going straight from road to road|
|turning left from road to road|
|going straight from road to road|
|turning left from road to road|
Traffic Arrival Process: Vehicles arrive at road entrances randomly and select a route in advance. All arrivals follow the same Bernoulli process (an approximation to Poisson process) but with different rates , where is route index, . For example, a vehicle following route will arrive at entrance of road with probability each second. To simulate heterogeneous traffic demands, we set roads to be busy roads and roads to be less busy roads. Specifically, . All vehicles enter one road from random lanes.
Traffic Signal Timing: Rules for actuating traffic signals have been introduced in Agent Action of Section II and examples are given in Fig. 4 and Fig. 5. Here, we set green light interval seconds and yellow light interval seconds.
Agent Parameters: The agent is trained for episodes. Each episode corresponds to traffic of hours. For -greedy method in Algorithm 1, parameter is set to be for all episodes. Set discount factor , update rate , learning rate of RMSProp algorithm to be and capacity of replay memory to store data for episodes.
Simulation data processing: Define the delay of a vehicle at an intersection as the time interval (in seconds) from the time the vehicle enters one road of the intersection to the time it passes through/leaves the intersection. From the definition, we know that vehicle staying time is closely related to vehicle delay. During simulations, we record two types of data into separate files for all episodes: the sum of staying time of all vehicles at the intersection at every second and the delay of vehicles at each separate road . After collecting these data, we calculate their average values for each episode.
Iv-B Simulation Results
First, we examine simulation data to show that our algorithm indeed learns good action policy (i.e., traffic signal control policy) that effectively reduces vehicle staying time , thus reducing vehicle delay and traffic congestion, and that our algorithm is stable in making control decisions, i.e., not oscillating between good and bad action policies or even diverging to bad action policies.
The average values for the sum of staying time of all vehicles at the intersection are presented in Fig. 9. From this figure, we can see that the average of the sum of vehicle staying time decreases rapidly as the agent is trained for more episodes and finally reduces to some small values, indicating that the agent does learn good action policy from training. We can also see that after episodes, average vehicle staying time keeps stable at small values, indicating that our algorithm converges to good action policy and algorithm stabilizing mechanisms, experience replay and target network, work effectively.
The average values for delay of vehicles at each separate road are presented in Fig. 10. From this figure we see that average vehicle delay at each road is reduced greatly as the agent is trained for more episodes, indicating that our algorithm achieves adaptive and efficient traffic signal control. After the agent learns good action policy, average vehicle delay reduces to small values (around seconds for road , seconds for road , seconds for road and seconds for road ) and stays stable thereafter. From these stable values, we also know that our algorithm learns a fair policy: average vehicle delay for roads with different vehicle arrival rates does not differ too much. This is because long vehicle staying time, thus vehicle delay, at any road leads penalty to the agent (see (3)), causing the agent to adjust its action policy accordingly.
Next, we compare the vehicle delay performance of our algorithm to that of another two popular traffic signal control algorithms, longest queue first algorithm (turning on green lights for eligible traffic with most queued vehicles)  and fixed time control algorithm (turning on green lights for eligible traffic using a predetermined cycle), under the same simulation settings in IV-A. However, we change vehicle arrival rates by a parameter as , , during simulation, where values of , , are given in Section IV-A. Simulation results are summarized in Fig.11.
From Fig. 11, we can see that for busy roads , the average vehicle delay of our deep reinforcement learning algorithm is the lowest all the time: up to reduction when compared to fixed time control algorithm and up to reduction when compared to longest queue first algorithm. As traffic demand increases (i.e., as increases), average vehicle delay of fixed time control algorithm increases exponentially. This is because fixed time control algorithm is blind thus not adaptable to real-time traffic demands. Longest queue first algorithm can adapt to real-time traffic demand somewhat. However it only considers halting vehicles in queues, vehicles not in queues but to come soon are ignored, which is also useful information for traffic signal control. Our algorithm considers real time traffic information of all relevant vehicles, therefore outperforms the other two algorithms. Another observation from Fig. 11 and Fig. 11 is that as traffic demand increases, the average vehicle delay of our algorithm increases only slightly, indicating that our algorithm indeed adapts to dynamic traffic demand to reduce traffic congestion. However, this comes at the cost of slight increase in average vehicle delay at less busier roads , as shown in the zoomed in portions of Fig. 11 and Fig. 11.
V Related Work
In this section, we review related work on adopting deep reinforcement learning for traffic signal control.
After formulating traffic signal control problem as a reinforcement learning problem, Li et al. proposed to use deep stacked autoencoders (SAE) neural network to estimate the optimal Q-values , where the algorithm takes the number of queued vehicles as input and queue difference between west-east traffic and north-south traffic as reward. By simulation, they compared the performance of their deep reinforcement learning algorithm to that of conventional reinforcement learning algorithm (i.e., without deep neural network) for traffic signal control, and concluded that deep reinforcement learning algorithm can reduce average traffic delay by . However, they did not detail how target network is used for Q-value estimation nor how target network parameters are updated, which is important for stabilizing algorithm. Furthermore, they simulated an uncommon intersection scenario, where turning left, turning right are not allowed and there is no yellow clearance time. Whether their algorithm works for realistic intersection remains unknown. Different from this work, our algorithm does not use human-crafted feature, vehicle queue length, but automatically extracts all useful features from raw traffic data. Our algorithm works effectively for realistic intersections.
Aiming at realistic intersection, Genders et al.  also proposed a deep reinforcement learning algorithm to adaptively control traffic signals, where convolutional neural networks are used to approximate optimal Q-values. Their algorithm takes vehicle position matrix, vehicle speed matrix and latest traffic signal as input, change in cumulative vehicle delay as reward, and uses a target network to estimate target Q-values. Through simulations, they showed that their algorithm could effectively reduce cumulative vehicle delay and vehicle travel time at an intersection. However, a well known problem with deep reinforcement learning is algorithm instability due to the moving target problem as explained in . The authors did not mention how to solve this problem, a major drawback of their work. Furthermore, they did not consider fair traffic signal control issues as they mentioned and their intersection model does not have left-turn waiting areas, which is a commonly adopted and efficient mechanism for reducing vehicle delay at an intersection. In comparison, our algorithm not only improves algorithm stability but also finds fair traffic signal control policy for common intersections with left-turn waiting areas.
Pol addressed the moving target problem of deep reinforcement learning for traffic signal control in  and proposed to use a separate target network to approximate the target Q-values. Specifically, they fix target network parameters for time steps during training, however update DNN network parameters every time step and copy DNN parameters into target network parameters every time steps (referred to as hard update). By simulation, they showed that algorithm stability is improved if is set to be a proper value neither small nor large. However, this proper value of cannot be easily found in practice. Moreover, they used inefficient method to represent vehicle position information, which results in great computation cost during training. Specifically, the author used a binary position matrix: one indicating the presence of a vehicle at a position and zero indicating the absence of a vehicle at that position. Instead of covering only the roads area relevant to traffic signal control, they set the binary matrix to cover a whole rectangular area around the intersection. Since vehicles cannot run at areas except roads, most entries of the binary matrix are zero and redundant, making the binary matrix inefficient. Differently, our algorithm solves moving target problem by softly updating target network parameters , not needing to find proper value of . Moreover, our algorithm represents vehicle position information efficiently (vehicle position matrix only covers intersection roads) thus reducing training computation cost.
We proposed a deep reinforcement learning algorithm for adaptive traffic signal control to reduce traffic congestion. Our algorithm can automatically extract useful features from raw real-time traffic data, which uses deep convolutional neural network, and learn the optimal traffic signal control policy. By adopting experience replay and target network mechanisms, we improved algorithm stability in the sense that our algorithm converges to good traffic signal control policy. Simulation results showed that our algorithm significantly reduces vehicle delay when compared to another two popular algorithms, longest queue first algorithm and fixed time control algorithm, and that our algorithm learns a fair traffic signal control policy such that no vehicles at any road wait too long for passing through the intersection.
-  D. Zhao, Y. Dai, and Z. Zhang, “Computational intelligence in urban traffic signal control: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 4, pp. 485–494, July 2012.
-  M. Alsabaan, W. Alasmary, A. Albasir, and K. Naik, “Vehicular networks for a greener environment: A survey,” IEEE Communications Surveys & Tutorials, vol. 15, no. 3, pp. 1372–1388, Third Quarter 2013.
-  A. A. Zaidi, B. Kulcsár, and H. Wymeersch, “Back-pressure traffic signal control with fixed and adaptive routing for urban vehicular networks,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 8, pp. 2134–2143, August 2016.
-  J. Gregoire, X. Qian, E. Frazzoli, A. de La Fortelle, and T. Wongpiromsarn, “Capacity-aware backpressure traffic signal control,” IEEE Transactions on Control of Network Systems, vol. 2, no. 2, pp. 164–173, June 2015.
-  P. LA and S. Bhatnagar, “Reinforcement learning with function approximation for traffic signal control,” IEEE Transactions on Intelligent Transportation Systems, vol. 12, no. 2, pp. 412–421, June 2011.
-  B. Yin, M. Dridi, and A. E. Moudni, “Approximate dynamic programming with recursive least-squares temporal difference learning for adaptive traffic signal control,” in IEEE 54th Annual Conference on Decision and Control (CDC), 2015.
-  I. Arel, C. Liu, T. Urbanik, and A. G. Kohls, “Reinforcement learning-based multi-agent system for network traffic signal control,” IET Intelligent Transport Systems, vol. 4, no. 2, pp. 128–135, June 2010.
-  P. Mannion, J. Duggan, and E. Howley, An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control. Springer International Publishing, May 2016, ch. Autonomic Road Transport Support Systems, pp. 47–66.
-  M. J. Neely, “Dynamic power allocation and routing for satellite and wireless networks with time varying channels,” Ph.D. dissertation, LIDS, Massachusetts Institute of Technology, Cambridge, MA, USA, 2003.
-  R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.
-  V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
-  W. Genders and S. Razavi, “Using a deep reinforcement learning agent for traffic signal control,” November 2016, [Online]. Available: https://arxiv.org/abs/1611.01142.
-  T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning, vol. 4, no. 2, 2012.
-  T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” February 2016, [Online]. Available: https://arxiv.org/abs/1509.02971.
-  D. Krajzewicz, J. Erdmann, M. Behrisch, and L. Bieker, “Recent development and applications of sumo simulation of urban mobility,” International Journal On Advances in Systems and Measurements, vol. 5, no. 3 & 4, pp. 128–138, December 2012.
-  R. Wunderlich, C. Liu, and I. Elhanany, “A novel signal-scheduling algorithm with quality-of-service provisioning for an isolated intersection,” IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 3, pp. 536–547, September 2008.
-  L. Li, Y. Lv, and F.-Y. Wang, “Traffic signal timing via deep reinforcement learning,” IEEE/CAA Journal of Automatica Sinica, vol. 3, no. 3, pp. 247– 254, July 2016.
-  E. van der Pol, “Deep reinforcement learning for coordination in traffic light control,” Master’s thesis, University of Amsterdam, August 2016.