[

Insects are a significant source of inspiration for developing novel solutions to autonomous flight of Micro Air Vehicles (MAVs). Even with limited computational and energy resources, insects are able to effectively complete challenging tasks with relatively complex behaviors. One behavior that is of particular interest, is landing.

Insects have been shown to primarily rely on visual inputs when landing, whereby many insects regulate a constant rate of expansion (or divergence) of optical flow [Gibson1950] to perform a smooth landing [Baird2013]. This approach has inspired some robotic implementations of constant divergence landing strategies [Herisse2012]. Although simple in concept, this approach is quite difficult to implement in reality. A direct implementation causes the robot to become unstable as the vehicle nears the ground. This is a result of the non-linear interaction between the vehicle control and sensing, in the presence of delay, measurement noise and environmental disturbance [Croon2016]. Different augmentations to the standard control scheme have been made, most, simply perform slow landings [Herisse2012], use long landing legs to delay the onset of this instability and touch the ground before they occur or by switching to alternative measures like time-to-contact [Clady2014, Kendoul2014] or velocity in-plane of the camera [Rodolfo2015, Miller2019]. More recently, there have been attempts to identify the instability and adapt the landing strategy by adjusting control gains [PijnackerHordijk2018, Ho2018].

An alternative to these manually designed approaches is to have an optimization technique automatically develop a suitable solution that is robust to these instabilities. This optimization may reveal new solutions to this problem, and perhaps even alternative hypotheses on what flying insects such as honeybees may be doing. Some attempts have been made to do this [Howard2016] but to the best knowledge of the authors, none of these have been implemented on real world robots.

This is due in part, to the effects of the differences between the simulated environment, commonly used for behavioral optimization, and the real world. This resultant difference in the robotic behavior is commonly referred to as the reality gap [jakobi1995noise, Nolfi2000, Bongard2013]. Several approaches have been used to make controllers more robust to the reality gap with the most significant being adding appropriate and varied noise [Jakobi1995], co-evaluation of controllers in the real world to test transferability [Koos2010] and inspired by conventional control theory, there are approaches utilizing abstracted outputs from the neurocontroller with a closed loop control system to actively reduce the effect of reality gap [Szczawinski2013, Scheper2016a].

In this paper, we describe a method to optimize a quick but safe landing maneuver for a quadrotor MAV equipped with a downward facing camera. The neurocontrollers are given only the divergence of the optical flow field from the camera as input and its time derivative. Although this abstracted input may reduce the possible behaviors the controllers can express, building on the work in [Scheper2017], we will demonstrate that this abstraction leads to a robust transfer from simulation to reality after virtual optimization.

The following consists of a summary definition of optical flow in Section 2. The flight platform and simulation environment is then described in Section 3. Next, the performance of conventional constant divergence landing approaches are presented in Section 4 to provide some baseline performance to compare the optimized policies against. The evolutionary setup and neural models used for the neurocontrollers is then described in Section 5. This is followed by a presentation and analysis of the optimized policies in Section 6. The reality gap and the results from the real world experiments are presented in Section 7 and Section 8 from which we draw some conclusions in Section 9.

To perform optical flow based landing as in [Croon2013a, Ho2016, PijnackerHordijk2018, Ho2018], we must first define the optical flow parameters. The formulation here is a summarized version of that presented in [Croon2013a]. This algorithm provides a good trade-off of accurate optical flow estimates while using relatively limited computational resources. This allows the perception and control loop to operate at high frequency and low latency on the embedded flight platform used in this paper. Alternative optical flow estimation methods could be used given that the estimation runs fast enough to facilitate the flight control. An investigation of the accuracy and reliability of the optical flow estimation is out of the scope of this paper.

If we assume that we have a downward-looking camera overlooking a static planar scene, as shown in Fig. 1, we can derive the perceived optical flow as the result of the camera ego-motion. The derivation of this optical flow model relies on the two reference frames, the inertial world frame is denoted by $W$ and the camera frame centered at the focal point of the camera denoted by $C$. In each of these frames, position is defined through the coordinates ($X$, $Y$, $Z$), with ($U$, $V$, $W$) as the corresponding velocity components. The orientation of $C$ with respect to $W$ is described by the Euler angles $\varphi$, $\theta$, and $\psi$, denoting roll, pitch, and yaw, respectively. Similarly, $p$, $q$, and $r$ denote the corresponding rotational rates.

In this work, we used a FAST corner detector and a Lukas–Kanade tracker implemented using OpenCV as in [Ho2018], we refer the reader there for more details. Throughout this paper, $N$ is limited to 100 if there were more than 100 points or simply all points if fewer have been tracked.

The flight platform used in this paper is the Parrot Bebop 2 quadrotor MAV¹, a picture of this vehicle flying in our indoor test environment can be found in Fig. 2. This vehicle is equipped with a 780 MHz dual-core Arm Cortex A9 processor, forward and downward facing CMOS cameras, sonar, and barometer enabling autonomous flight capabilities for up to 25 minutes. Full 3D flight control is enabled with the onboard use of the open source PaparazziUAV autopilot software [Hattenberger2014]. In this work, we extract global optical flow from the downward facing camera using the Lucas–Kanade optical flow method executed onboard the vehicle [Lucas1981]².

Throughout the paper, an Optitrack³ motion capture system was used to measure a ground truth of the vehicle position and motion. As the control task in this paper is only in the vertical axis, this ground truth measure was communicated to the vehicle to facilitate the control of the lateral axis of the vehicle. This was however not used in the vertical loop where instead the optical flow estimated onboard the vehicle was used.

To optimize our behavior in simulation, we first need a model of the vehicle. To this end we use a simple dynamical model of the vehicle, which is restricted to vertical motion only. The thrust generated by the rotors is modeled as a first order response with the dynamics defined in (7).

where $T_{sp}$ is the thrust set-point and spin-up spin-down time constant ${\tau }_T$ has a nominal value of 0.02 s. The thrust output is limited in the range $[$-0.8$\cdot$g, 0.5$\cdot$g$]$ as a conservative model the maximum acceleration of the real vehicle.

The model used to describe the divergence estimation is based on the work presented in [Ho2016]. The observed divergence is the result of adding latency to the true divergence along with two types of noise, simple white noise drawn from $N$(0,${\sigma }_w^2$) and an additional noise proportional to the divergence magnitude drawn from $N$(0,${\sigma }_p^2$). [Ho2016] identified typical values for ${\sigma }_w$ and ${\sigma }_p$ as 0.1 s${}^{-1}$ and the latency $L$ in the range of [50, 100] ms. We use similar nominal values for the standard deviations ${\sigma }_w$ and ${\sigma }_p$ are 0.1 s${}^{-1}$ and an exaggerated range for the latency $L$ in the range of [1,4] samples or [20, 133] ms. Some additional computational jitter has been included, this simulates the situation of missed frames which happens when there are either insufficient or too many image features to be tracked. The chance of a missed frame is randomly determined with a given probability held constant for each simulation run randomly drawn from the range [0, 0.2].

Before we start to optimize a divergence based landing controller, let us first investigate the naive approach of a constant gain, constant divergence landing as studied in [Herisse2012, VanBreugel2014, Croon2016].

Fig. 3 shows the time history of a two constant divergence landing controllers performing a landing with a divergence set-point of 0.5. Controller $C_1$ has a relatively high gain and $C_2$ a low gain. The controllers are activated 1 s after the simulation starts. Fig. 17 shows the steady-state response of the controller with the time history control signal super-imposed.

These plots show that controller $C_1$ quickly reaches the desired divergence but as the vehicle descends the controller becomes increasingly unstable. This instability is due to delays in the sensing and control loop and is well described in [Croon2016]. In contrast, $C_2$ does not quickly achieve the desired divergence due to the low gain in the controller. Additionally, due to the non-linear relationship between the divergence and acceleration, the vehicle overshoots the desired set-point before finally slowing. Despite this poor tracking performance, the low gain does delay the onset of instability observed with $C_1$, resulting in a rather smooth landing profile.

Every evolutionary process is defined by: a population of candidate individuals, each with a given policy which must be evaluated; a way to evaluate these policies; a selection mechanism to filter out bad policies; and a method to alter the individuals to generate new policies. Here, we use a mutation-only evolutionary algorithm similar to ($\mu$ + $\lambda$) approach, where a population is maintained from which offspring are generated using a mutation operator. Offspring that are better than members of the population replace these members. The difference with the standard implementation is that we retest the current population on a random set of simulation parameters with every generation, rather than once as is commonly done. With the high level of non-determinism in our simulated environment, this ensures no individuals are preferred simply because they were tested on an easy set of conditions.

The evaluation of the individuals is done by simulating the policy on four independent simulation runs, initializing the vehicle at a standstill from four different altitudes, namely 2, 4, 6 and 8 m. The simulation ends when the vehicle exceeds 15 m above the ground, gets within 5 cm of the ground or exceeds 30 s of simulation time.

We use a multi-objective approach here, where the individuals must minimize three fitness functions measured at the end of a simulation: the total time to land ($f_1$); the final height ($f_2$); and the final velocity ($f_3$). NSGA-II is used to perform a non-dominated sorting of the population and determine which individuals are better than others in this multi-objective framework [Deb2002].

All the simulation parameters mentioned in Section 3 are randomly perturbed and set at the start of a generation. The ranges of the evolutionary parameters used in this work are summarized in Table 1. Altogether, this should encourage the optimization to develop a policy that can reliably land quickly and safely from different altitudes. This is all developed using the DEAP framework [DEAP_JMLR2012] with multi-threaded implementation utilizing scoop in Python⁴⁵.

The input to the neurocontroller is the simulated divergence and the derivative of the divergence $\Delta D=\frac{D_t-D_{t-\Delta T}}{\Delta t}$. The output was used to control the thrust of the vehicle leading to an acceleration. We utilized three types of neural networks to investigate the effect of recurrent connections on the evolved solution. We implemented a feed-forward neural network (hereafter referred simply as NN), a recurrent neural network (RNN) and a continuous time recurrent neural network (CTRNN). All networks had three layers, the first with 2 neurons, the hidden layer with 8 neurons and the output with 1 neuron.

The use of the non-dominated sorting and selection, results in the genetic optimization spreading the population of policies over the pareto front of the fitness landscape. As such, to evaluate the performance of the optimization, it is sensible to look at the ratio of the encapsulated volume and the area of the pareto front ($\nu =volume/area$) as shown in Fig. 4. This figure shows that the performance generally improves before leveling off after about 150 generations. This performance is also mostly stable as the performance remains flat after 150 generations. This also shows that this trend is consistent over the multiple initializations of the optimization and over the different types of neural architecture.

Fig. 5 shows the accumulated Pareto front of all individuals from the different runs of the optimization. This figure only shows the performance on the touchdown velocity and the time fitness as the fitness based on the height was consistently minimized for all individuals. As such, all optimized policies converged to perform the desired landing task. This figure shows how the policies are spread over the inherent trade-off of reliable quick vs soft landing. The performance seems not to be strongly correlated with the neural architecture as all three types are represented in the pareto front. Additionally, the three architectures seem well distributed.

To investigate the sensitivity of the performance to different environmental conditions, we subjected the individuals of the pareto front to a validation test with 250 simulations while varying the environmental settings of the run. All individuals where subjected to the same set of conditions to make for a fair comparison. Fig. 6 shows that the individuals that optimized to have slow and soft landings have a small spread in the fitness performance whilst individuals that were optimized to faster landings have a larger variation in landing speed. This suggests that the policies that perform faster landings may show oscillatory behavior causing their touchdown speed to vary depending on the sensor noise or environmental perturbations. We will investigate this more below.

As it is not feasible to analyse them all, three individuals from the each architecture are be selected for further analysis of the landing behavior. The remainder of this section will dive deeper into the types of behaviors optimized by the different neurocontrollers.

Everything discussed so far has been in a simulated environment, one that is a significant simplification of reality to facilitate high speed evaluation of the neurocontrollers. As we move to the real world, we can therefore expect various differences leading to a reality gap. This section aims to identify some of these differences.

Let us first look at the control of the vehicle, in simulation, the output of the neurocontroller was acceleration, which after being fed through a low pass filter to simulate the spin-up of the rotors was implemented by the simulated rotors. In reality, this desired acceleration must first be converted to a thrust command for the rotors. If we use a naive approach here, we can simply determine a linear transform from desired acceleration to thrust, the results of which are shown in the top plot of Fig. 10. This figure shows a set of real world neurocontroller landings using a constant scaling factor for desired acceleration to thrust. As the vehicle starts to move the command tracking is good but as it starts to descend the tracking degrades. This is almost certainly due, in part, to the unmodeled drag and non-linear aerodynamic effects of descending through the downwash of the propellers.

This poor tracking leads to a noticeable reality gap in the landing performance. This can be reduced with the use of a closed loop controller as proposed in [Scheper2016a], instead of a linear transform. The results using a Proportional-Integral (PI) controller, minimizing the error between the commanded and measured acceleration on the vehicle, are shown in the bottom plot of Fig. 10. The controller effectively abstracts away from the raw motor commands to a desired acceleration, which significantly improves the tracking performance allowing us to cross this reality gap.

The goal of this paper is to highlight the use of abstracted inputs to improve the robustness of optimized policies to differences in the input. To do this we will investigate the performance of the vehicle with the use of two different types of cameras with significantly different divergence signal output performance characteristics.

The first camera uses the monochromatic information from the bottom looking CMOS camera built into the Parrot Bebop 2 using the size divergence estimation method described in [Ho2018]. The second camera is the Insightness SEEM1 Dynamic Vision Sensor (DVS) using the efficient plane fitting optical flow estimation technique described in [PijnackerHordijk2018]. This event based camera does not generate frames as a conventional image sensor but rather measures logarithmic light changes at each pixel independently and asynchronously. This makes the sensor conditioned to operate with high speed motion with low latency and relatively low data throughput. A simple comparison of the difference in data from the frame-based and event-based data can be seen in Fig. 11. This type of camera has been previously used to facilitate high speed landings as shown in [PijnackerHordijk2018]. A schematic showing the optical flow processing pipeline for the CMOS and event-camera can be found in Fig. 12.

Fig. 13 shows a comparison of the divergence estimation error from these two cameras highlighting how different the output of these camera is and how different they both are to the camera statistics used in simulation. Some additional differences in the camera properties are summarized in Table 2. We will investigate in the next chapter if these differences result in a significant reality gap.

Using the closed loop PI controller to control the thrust as described in the previous section, we performed a set of landings with some of the neurocontrollers identified in Section 5. All flights were initiated from a steady hover at an altitude of 4 m. Fig. 14 shows the results from the controllers NN${}_1$ and NN${}_2$, Fig. 15 shows the results from RNN${}_1$ and RNN${}_2$ and Fig. 16 shows the results from CTRNN${}_1$ and CTRNN${}_2$. These results are plotted for both the CMOS and DVS cameras to see how well these two sensors affect the landing profile. The results from simulation have also been plotted to see how well the real world performance fits with the simulated, a measure for the eventual reality gap. NN${}_3$, RNN${}_3$ and CTRNN${}_3$ were not tested in reality as their relatively high touchdown velocity in simulation may cause damage to our real world vehicle. This is in-fact a benefit of the multi-objective optimization scheme used here, the user can simply choose a policy that performs the trade-off of the fitness functions as desired.

In spite of the differences in the generation of the divergence between the two cameras, the landing performance is very similar. These two systems are also so similar to the simulated landing that the plot is hardly visible. This would suggest that the eventual reality gap is small despite significant differences in the way the input was generated.

Also notable is the repeatability of the landing maneuvers. The landings were performed three times each and each landing resulted in very similar trajectories. This shows that the evolutionary optimization converged to a robust solution as suggested by the analysis in Section 5.

This paper investigated the influence of abstraction of the sensory input to the reality gap for automatically optimized UAV agents tasked with performing quick yet safe landing. We have shown over multiple evolutionary runs and neurocontroller architectures, that abstraction does not unduly hamper the optimization power of the optimization as the agents developed a robust and effective method to land.

The optimized agents showed some landing strategies that were before not imagined by the human designers. One notable strategy is that instead of a simple proportional controller for the entire state space, an asymmetric response may be more appropriate to delay the onset of oscillations when performing the landing procedure. A strong response when the divergence error is positive and a weaker response when negative seems a good approach.

Tests in the real world showed the presence of significant differences between simulation and reality. The most significant was that the acceleration command tracking performance was poor, likely due to the drag and other non-linear aerodynamic effects which were not considered in simulation due to their modeling complexity. The resultant reality gap was crossed with the use of a closed loop controller representing another layer of abstraction, from the low-level raw motor control values to a desired vertical acceleration, which ensures robustness to the real world uncertainty.

Finally, we showed that abstraction on the sensory input of the neurocontroller was robust to the reality gap when using two different input estimation techniques. Although two cameras with different imaging techniques were used, the resultant landing profile was very similar. Abstraction can therefore be a powerful tool when crossing the reality gap.

Future work could investigate how this approach can generalize to different vehicles operating in different real world environments. This would help to demonstrate the effectiveness of the approach to not only cross the reality gap but to address the more general transfer problem.

We would like to thank the team at Insightness AG for their assistance in porting the SEEM1 SDK to operate on the ARM processor of the Parrot Bebop 2. Without their assistance the results in the paper would not be possible.