Tackling Occlusions & Limited Sensor Range with Set-based Safety Verification
Provable safety is one of the most critical challenges in automated driving. The behavior of numerous traffic participants in a scene cannot be predicted reliably due to complex interdependencies and indiscriminate behavior of humans. Additionally we face high uncertainties and only incomplete environment knowledge. Recent approaches minimize risk with probabilistic and machine learning methods – even under occlusions. These generate comfortable behavior with good traffic flow, but cannot guarantee safety of their maneuvers. Therefore we contribute a safety verification method for trajectories under occlusions. The field-of-view of the ego vehicle and a map are used to identify critical sensing field edges, each representing a potentially hidden obstacle. The state of occluded obstacles is unknown, but can be over-approximated by intervals over all possible states. Then set-based methods are extended to provide occupancy predictions for obstacles with state intervals. The proposed method can verify the safety of given trajectories (e.g. if they ensure collision-free fail-safe maneuver options) w.r.t. arbitrary safe-state formulations. The potential for provably safe trajectory planning is shown in three evaluative scenarios.
Index Terms— ADAS, automated vehicles, formal verification, reachability analysis, risk assessment, occlusions, field-of-view.
Over the last decades research effort about ADAS and fully automated vehicles has increased drastically in academia and the commercial sector. The latter presenting a worrying push to early market introduction as at least two major challenges, safety and scalability, have not been solved yet .
Scalability is an issue, because most state of the art approaches require sensors and processing power that is not, nor will likely be available for prices that allow mass sales in the near future .
Provable safety on the other hand is a must in regards to accountability, user acceptance and in consequence as a prerequisite for legal permission. As extensively explained in , provable safety is of special interest when applying machine learning approaches as they often lack formal validation methods. Therefore they introduce the notion of blame, define a concept of Responsibility Sensitive Safety (RSS) and explain how to develop automated vehicles that provably fulfill RSS. Another promising and meanwhile complementary approach to safety verification is the computation of reachable sets of obstacles (by physics and traffic law limited reachable states) . These are used to prove if the ego trajectory allows a fail-safe maneuver in the next planning step and, therefore, is safe in itself. They formally introduce the method for sets of initial obstacle states, but de facto developed only over-approximations for exactly known initial obstacle states.
In reality we encounter a variety of uncertainties and incomplete environment knowledge. The former arise in all stages of an ADAS pipeline, from measurement noise and sensor limitations, over processing steps as localization, tracking, prediction and planning (due to modelling errors or unexpected situations), up to the imprecise realization of planned trajectories. Additionally the environment knowledge is incomplete because of limited perception range and due to occlusions from static and dynamic obstacles alike.
Ii Related Work
A lot of research has been done to approach the problem of risk assessment. A well-arranged survey is given in .
Most of the methods presented try to develop behavior models and then check for collisions under the assumption of those models or detect deviations from these models. Either way such risk assessment approaches assume that all possible maneuvers in a given scenario can be modeled or that unexpected situations can be detected reliably. Additionally many do not consider the incompleteness of an environment model at all.
Recently focus on occlusion-aware111We will use “occlusion” for any area outside of our field of view. This includes areas occluded by obstacles, but also areas outside the sensor range. risk assessment and behavior generation has increased [4, 5, 6, 7, 8, 9, 10, 11]. These range from simple visibility modeling that improves the tracking of previously detected obstacles , to sophisticated multi-layered environment models .
Some of them explicitly consider uncertainties [11, 4, 7], while others use visibility analysis to define velocity constraints [6, 5]. Only two publications among those prove at least passive motion safety222“If a collision takes place, the robot will be at rest.”   or prove collision freedom at discrete time steps of their trajectories333Collisions between discrete time steps will not be detected here. .
A promising approach is given by . They model and predict the environment with three grid map layers: Object-based, object-free and unobservable environment. During planning they treat a cell as occupied as soon as one of the layers is predicted as occupied.
Still, these methods minimize the risk of collisions at most [7, 6, 4, 5] and give no or too weak [10, 11] safety guarantees. Some of the earliest approaches show even fundamental problems like ignoring not yet observed obstacles , analyzing occluded areas without prediction  or only investigating occlusions from static obstacles .
Concluding, all these methods lack verification of higher levels of safety444Meaning higher than passive or even passive friendly safety , but at least RSS . considering limited environment knowledge.
Vehicles with such risk assessment strategies will try to follow their intended behavior as long as no noteworthy risk is detected. But this could still lead to situations where a collision is inevitable.
We want to promote an inverse methodology. An automated vehicle should only follow its intended trajectory as long as it can prove that it is safe. Let us ensure safety first and then increase comfort and traffic flow, not the other way around. Such a development approach will lead to more conservative behavior in the first years, but on the other hand help in earlier legal permission and to gain users confidence.
Thus we tackle the safety verification problem in scenarios with occlusions by over-approximating all possible states instead of engineering discrete maneuvers or maneuver classes.
Hence our safety concept does not rely on e.g. the performance of an intention estimation module. It depends only on the reliability of an ego localization555w.r.t. the map., the detection of obstacles, of occluded areas and on a map666Featuring the road topology, geometry and traffic rules..
Therefore in this work we characterize potential risk from occlusions and limited sensing (section III), enhance the reachable set over-approximations introduced by  to serve well for these critical perception field bounds (section IV) and show its potential for safe trajectory planning (section V) in several simulative scenarios (section VI).
The approach is not only applicable for fully automated vehicles as in our evaluation, but also for level 1–4 ADAS , e.g. collision warning systems. Uncertainties from measurements can easily be incorporated as well as any safety definition, because we use a generic safe-state formulation.
Iii Risk from Occlusions and Limited Sensing
As a necessary preparation for the following chapters, we first characterize the potential risk that results from occlusions and limited sensor range.
We model the environment mostly as in , meaning that the road geometry and topology is given as a lanelet map , a lane consists of consecutive lanelets, the ego vehicle and other (visible) obstacles have rectangular shape and the over-approximations of predicted occupancies are modeled with polygons. The main difference in modeling is that we explicitly represent critical sensing field borders.
To do so we need a representation of the sensing field of the ego vehicle. This can be provided by accurate sensor models or explicitly mapped from sensor data. The mapping is straight forward for 3D range sensors, e.g. all range measurements (occupied or free) can be modeled as rays with specified beam angle and used directly as geometrical shapes or transferred into a visibility layer of a grid map at the cost of discretization errors.
In our simulative evaluation we use a simple visibility model, assuming a range sensor with viewing range mounted on top of the vehicle center using a direct geometrical representation, see the light blue filled area in Fig. 2.
The borders of the sensing field can then be extracted and intersected with all lanelets. Each of these intersections generates at least one border segment and can be classified into relevant and irrelevant sections. For computational efficiency we assume the border to be modeled as a polygon, thus the border sections consist of line segments.
The classification of relevant sections is not self-evident, but can be reduced to false positives only, which are not critical. In general only sections that could hide obstacles with right of way need to be examined. Falsely considering nonrelevant sections as relevant comes at an additional performance cost, but does not affect the safety verification result. That is because the occupancy of corresponding potential obstacles does not intersect with the current or any of the coming ego lanelets.
Employed on the lanelet representation, relevant sections can be classified as nonrelevant without false-positives in these incomplete cases:
Border segments that do not intersect any lanelet which can lead to or cross any of the lanelets the ego vehicle is currently or will be traveling.
Segments on lanelets with right of way.
Segments in the ego lane that are behind the ego vehicle.777 This can be motivated by the concept of blame  or similar, as the blame of an accident is on the rear car as long as the ego did not cut in the others lane with a unsafe longitudinal distance.
Edges that lead outside of the sensing field888w.r.t. the driving direction..
For multiple segments on the same side999w.r.t. intersection. of the same lane only the foremost segmentfootnote 9 is relevant.
See Fig. 2 for a descriptive example of critical sensing field edges.
Having identified the critical sensing field edges, the key question arises: What is the potential risk that results from these? The illustrative answer is that each occluded area might contain at least one obstacle with unknown state. A naive approach of simply spawning countless virtual obstacles with randomly chosen possible states and predict their occupancy is not an option as a stochastic approach will not enable us to verify planned trajectories with reasonable computational cost. But even though the state is unknown, we can over-approximate possible states as bounded sets, such that other states are either physically impossible or would not lead to an accident of our blame (as for tremendous speeding of the obstacle for example). Therefore we define one virtual obstacle for each critical edge with the following state set, described as intervals in orientation and velocity and a line segment in the initial position defined by the two edge vertices and . We use the following notation:
Iv Reachable Sets for Intervals of Initial State
In this section we derive reachable set over-approximations for obstacles with initial state intervals, as in creftype 1. These are based on the and over-approximations from . describes the physically reachable area based on Kamm’s circle, limiting the possible absolute acceleration and prohibiting driving backwards as an assumption. To incorporate a lane-following property describes the longitudinally reachable area with maximum velocity and maximum engine power. The maximum velocity should be set to a realistic value which represents expectable speeding (e.g. of the speed limit), the maximum engine power can be set to infinity.
The following subsections expand the and formulas step by step to intervals of initial state. We keep the original notation wherever feasible, but redefine some variables to keep a clean notation, free of avoidable indices and accents.
Iv-a Acceleration-Based Occupancy
Similar to  we assume the initial obstacle state to be represented in local coordinates:
We also use their formulation of Kamm’s circle with center and radius and the function bounding that circle over time:
Please see Fig. 2(a) for a graphical representation.
Iv-A1 Interval of Initial Velocities
With an interval in the initial velocity , but known orientation and , we can define
and likewise. The occupancy of the obstacle for a time period of can then be over-approximated by the polygon spanned by the points :
as visualized in Fig. 2(a).
The left part is equivalent to the original , but vertices are computed using . This occupancy encloses all as each circle has the same radius , and center . Therefore it is enclosed by the polygon spanned from and . Consequently all with are enclosed by the polygon , proving that this polygon is an over-approximation of all reachable states for an obstacle with interval velocities.
Iv-A2 Interval of Initial Orientations
With additionally an initial orientation interval of the whole occupancy rotates around the origin depending on the real initial orientation of the obstacle. We can again over-approximate this set. The borders of the set can be derived by rotating counterclockwise to and clockwise to by . and over-approximating the circle given through the rotation of the furthest longitudinal point with
for integers . A reasonable approximation of the circle over of around can already be achieved with .
Fig. 2(b) illustrates the construction. The prove that each polygon with rotation is enclosed by the polygon
follows trivially from its construction.
Iv-A3 Interval of Initial Positions
The transfer to position intervals, meaning linear interpolations between both line segment vertices, can be realized by computing the occupancy for as described in the previous subsection, creating a duplicate that has been translated by and computing the convex hull over both occupancies .
The occupancy for each (each possible position on the line segment) is enclosed by the convex hull due to the linearity of translations and line segments. Thus is provably an over-approximation of the reachable set of an obstacle with an unknown but bounded initial state as in creftype 2.
The resulting over-approximation is visualized for realistic example parameters in Fig. 2(c).
Iv-B Lane-Following Occupancy
The adaption of the original formulation for interval initial states follows quickly. We use the same computation of shortest paths in lanes as , but choose the start- and end-point wisely from our interval. To do so, we first sort both vertices of the line segment w.r.t. the longitudinal path coordinates of the corresponding lane or lanelet and name them , such that without loss of generality. The start border of the lane occupancy is then given by . The maximal traveled distance can be obtained with the limited maximum speed and engine power model as in :
This enables us to compute the end border using the inflection point segmentation algorithm as .
Finally the occupancy prediction is obtained by computing and intersecting the reachable set approximations and for each critical sensing field edge in each prediction step period over the whole prediction horizon . A trajectory planner can use this prediction to verify if its intended trajectory is safe or not.
Note that in case of known initial state, where the intervals collapse to precise values , , , our formulation results in the same reachable set over-approximation as with the original method in .
V Trajectory Planning with Reachable Sets
The strategy for trajectory planning has been profoundly motivated and extensively explained in . Yet we want to briefly summarize the method and explain our proof-of-concept implementation to give a better understanding of the following evaluation.
The planning approach assumes that we start in a safe state with a reliable planning frequency of . The safety verification relies on induction. Starting from a safe state with a verified fail-safe trajectory at hands the planner will search for a desirable trajectory that will guarantee a fail-safe maneuver choice in the next planning step too. If it fails to find one, it switches to the fail-safe trajectory that has been found and verified in the previous planning step. Thus the vehicle will always follow a trajectory whose safety has been proven.
To do so a reference trajectory will be planned in each planning step , based on the current environment model. This might also contain reasonably good predictions of all obstacles, though it is not a prerequisite. But the better the prediction the higher the probability for a comfortable ride. As unexpected situations become rare events, the planner will be rarely forced to switch to the fail-safe maneuver.
In order to generate safe trajectories a potential trajectory is computed. It consists of an intended part, the first part of the reference trajectory over , and a following fail-safe part over towards a safe state. If we can prove that the potential trajectory does not intersect with any of the reachable set over-approximations of other obstacles, it is verified as safe. If the verification fails, the intended trajectory will be iteratively adapted until a verifiably safe trajectory, in the worst case the fail-safe trajectory of the previous planning step , is found.
Our proof-of-concept planner initializes the reference trajectory according to the intelligent driver model  along the centerline of the ego lane. If the verification fails the intended acceleration is gradually changed towards the fail-safe trajectory until the potential trajectory can be verified as safe.
The planning and prediction horizon is defined by the time needed to reach a safe state. This raises two questions: What is a safe state? And in case we need to decide for a fail-safe maneuver, do we want to reach the safe state fast or comfortably?
The safe state itself depends on the intended maneuver. As long as we don’t want to cross or intersect with other lanes a full stop can be seen as safe state in urban areas. But as soon as we merge into or cross the traffic of another lane we need to ensure a safe cut-in time. Hence a full stop is not a safe state in these cases. For a further discussion on safe states please refer to e.g. .
Choosing the desired trajectory towards a safe state is a trade-off between the comfort during a fail-safe maneuver and the probability of having to switch to one. Comfortable trajectories i.a. have low jerk and acceleration such that they need a long time to reach the safe state. This durations then defines the required prediction horizon for safety verification. With longer prediction horizons prediction occupancies are bigger, such that it becomes more likely that replanning is necessary to find a trajectory that succeeds in verification. As a consequence, if the prediction horizon is too long due to too comfortable fail-safe trajectories, the probability of too conservative behavior (from a users’ or traffic participants’ perspective) will be high. In our evaluation we used a compromise desired fail-safe deceleration of .
Vi Results and Evaluation
We evaluated our trajectory validation method on three critical urban scenarios with occlusion from static or dynamic obstacles and show its usability for safe trajectory planning. All scenarios represent or are inspired by real intersections in the city of Karlsruhe and a small town Fürstenfeldbruck. They are based on the commonroad scenarios GER_Ffb_1 and GER_Ffb_2 . Our modified versions have been contributed to the commonroad benchmark as scenarios GER_Ffb_1c, GER_Ffb_1d and GER_Ffb_2b.
We assume perfect perception from a range sensor with viewing range mounted on top of the vehicle center. The desired ego velocity is set to with a maximum comfortable acceleration of and maximum possible acceleration of , a realistic value for dry asphalt . For an easier understanding of the effects of occupancies, we do not adapt the desired velocity based on curvature. The prediction and planning horizon is defined by a desired fail-safe deceleration of , a compromise between comfortable fail-safe trajectories and keeping the desired driving speed. Other occupancy prediction parameters are , , , , , , .
Vi-a Merging on a T junction, static occlusion
The first scenario GER_Ffb_2b is a T-junction with a major road leading from east to west with a speed limit of and a minor road from the north. One edge of this intersection is occupied by a container, which we doubled in size to dramatize the occlusion effect.
The ego vehicle drives from the north and want to merge the major road to the west. The container occludes the eastern arm of the intersection, such that this arm will not be visible enough until around before merging into the lane.
The first row in Fig. 4 shows the behavior without incorporating occupancies, only based on occupancy predictions of visible obstacles. As to expect, the ego vehicle does not reduce its velocity in the first , then detects the obstacle and has to decelerate at maximum rate, as it does not find a safe trajectory any more. But the emergency braking comes too late and a collision is unavoidable if the other vehicle does not react or lacks enough reaction time, e.g. with a speeding of or with bad friction, e.g. because of a wet road.
The effectiveness of an occlusion aware occupancy prediction with our provided method is shown in the second row of Fig. 4. The ego vehicle reduces its velocity as soon as the potential trajectory cannot be verified as safe, because it intersects the occupancy of a potentially occluded obstacle. As in fact there is an obstacle appearing behind the occlusion, the ego vehicle safely comes to a full stop to give way. Without an obstacle it could safely continue merging after it has decelerated to around at the point to see far enough to the east.
Vi-B Crossing an X junction, static occlusion
The second scenario, shown in row 3 of Fig. 4 is an intersection in the residential area of Fürstenfeldbruck with a speeding limit of . We model it as an uncontrolled intersection, meaning without traffic lights or stop lines and that the priority-to-the-right rule applies, with .
The ego vehicle comes from the south and wants to cross the intersection. Similarly to the previous scenario, a static obstacle, in this case a residential building, occludes the easterly lanes.
The simulation without dynamic obstacles shows that the ego vehicle has to slow down significantly, in order to guarantee safety, but can finally pass the intersection. This is truly an appropriate behavior for such a dangerous setting as such intersections usually feature at least stop lines to force drivers to slow down and have a second look. The real intersection in Fürstenfeldbruck actually is even regulated by traffic lights.
Vi-C Turning at an X junction, dynamic occlusion
In the last scenario we use the road geometry from the second Fürstenfeldbruck intersection, but additionally incorporate a dynamic obstacle that passes from west to east.
The ego vehicle wants to cross the intersection to the west, so the dynamic obstacle occludes its view of the easterly lane. Again the question arises at what time and position it is safe to cross the intersection.
As can be seen in the last row of Fig. 4, the ego vehicle slows down more than without the dynamic obstacle due to the occlusion. Specifically this is the result of assessing every critical sensing field edge with the same prior, i.e. the same initial state intervals in velocity and orientation. It is apparent from the simulation sequence that the east lane is already partly visible before the dynamic obstacle occludes that area. As a consequence one could derive a better prior for this area based on those observations and continue driving earlier. We will briefly discuss this possible type of performance improvement in section VII.
Another observation in all scenarios is that the fail-safe trajectory has been activated even without dynamic obstacles. However a sophisticated planning method should slow down the vehicle earlier in order to optimize its approaching time and velocity such that the switch to a fail-safe maneuver will rarely be necessary.
Despite those inefficiencies the evaluation shows the need for occlusion-awareness in safety verification and thus highlights the value of our occlusion-aware occupancy prediction.
Vii Conclusions and Future Work
We motivate this contribution with the purpose of ensuring safety before comfort or traffic efficiency, especially when facing incomplete environment knowledge. We therefore characterize the risk from unperceived space as potentially hidden obstacles with an initial state that is unknown, but can be over-approximated with intervals. We enhance the reachable set approach introduced in  to predict occupancy over-approximations of such obstacles. To show the usefulness and potential of our approach, we implement a proof-of-concept planner which uses the occupancy prediction to plan provably safe trajectories. The performance is shown in three intersection scenarios with occlusions from static and dynamic obstacles. All collisions can be prevented while still moving through traffic fast enough.
Having a method to guarantee safety under occlusions w.r.t. an arbitrary safe-state, further work to improve comfort and efficiency can be done.
Comfort and traffic flow will increase with better prediction, because scenarios with low conflict probability can be approached and passed quicker. The fail-safe maneuver will still be guaranteed, but only a less comfortable one. On the other hand the vehicle could decelerate earlier when approaching intersections with high conflict probability.
Similarly also clever fail-safe velocity profiles can lead to a good compromise between comfortable, but not too conservative behavior.
Tracking of occluded areas would allow to reduce the initial state intervals of hidden obstacles by reasoning. Hence the predicted occupancies will be significantly reduced in dynamic scenarios, while still guaranteeing safety.
Finally, we will incorporate the proposed method in our prototype vehicle in the coming months and analyze the performance under real world conditions.
The authors thank Daimler AG for the fruitful collaboration and the support for this work.
-  S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a Formal Model of Safe and Scalable Self-driving Cars.”
-  M. Althoff and S. Magdici, “Set-Based Prediction of Traffic Participants on Arbitrary Road Networks,” vol. 1, no. 2.
-  S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion prediction and risk assessment for intelligent vehicles,” vol. 1.
-  S. Hoermann, F. Kunz, D. Nuss, S. Renter, and K. Dietmayer, “Entering crossroads with blind corners. A safe strategy for autonomous vehicles,” in 2017 IEEE Intelligent Vehicles Symposium (IV).
-  M. Lee, K. Jo, and M. Sunwoo, “Collision risk assessment for possible collision vehicle in occluded area based on precise map,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).
-  W. Chung, S. Kim, M. Choi, J. Choi, H. Kim, C. b Moon, and J. B. Song, “Safe Navigation of a Mobile Robot Considering Visibility of Environment,” vol. 56, no. 10.
-  S. Brechtel, T. Gindele, and R. Dillmann, “Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs,” in 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).
-  Miller Isaac, Campbell Mark, Huttenlocher Dan, Kline Frank-Robert, Nathan Aaron, Lupashin Sergei, Catlin Jason, Schimpf Brian, Moran Pete, Zych Noah, Garcia Ephrahim, Kurdziel Mike, and Fujishima Hikaru, “Team Cornell’s Skynet: Robust perception and planning in an urban environment,” vol. 25, no. 8.
-  M. Sadou, V. Polotski, and P. Cohen, “Occlusions in obstacle detection for safe navigation,” in IEEE Intelligent Vehicles Symposium, 2004.
-  S. Bouraine, T. Fraichard, O. Azouaoui, and H. Salhi, “Passively safe partial motion planning for mobile robots with limited field-of-views in unknown dynamic environments,” in 2014 IEEE International Conference on Robotics and Automation (ICRA).
-  W. Zhan, C. Liu, C. Y. Chan, and M. Tomizuka, “A non-conservatively defensive strategy for urban autonomous driving,” in 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).
-  K. Macek, D. A. V. Govea, T. Fraichard, and R. Siegwart, “Towards Safe Vehicle Navigation in Dynamic Urban Scenarios.”
-  S. O.-R. A. V. S. Committee and others, Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems. Technical Report J3016_201401.
-  P. Bender, J. Ziegler, and C. Stiller, “Lanelets: Efficient map representation for autonomous driving,” in 2014 IEEE Intelligent Vehicles Symposium Proceedings. IEEE.
-  M. Koschi and M. Althoff, “SPOT: A tool for set-based prediction of traffic participants,” in 2017 IEEE Intelligent Vehicles Symposium (IV).
-  M. Althoff and J. M. Dolan, “Online Verification of Automated Road Vehicles Using Reachability Analysis,” vol. 30, no. 4.
-  M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” vol. 62, no. 2.
-  M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Composable benchmarks for motion planning on roads,” in 2017 IEEE Intelligent Vehicles Symposium (IV).
-  C.-G. Wallman and H. AAström, Friction Measurement Methods and the Correlation between Road Friction and Traffic Safety: A Literature Review. Statens väg-och transportforskningsinstitut.