A Novel Disparity Transformation
Algorithm for Road Segmentation
The disparity information provided by stereo cameras has enabled advanced driver assistance systems to estimate road area more accurately and effectively. In this paper, a novel disparity transformation algorithm is proposed to extract road areas from dense disparity maps by making the disparity value of the road pixels become similar. The transformation is achieved using two parameters: roll angle and fitted disparity value with respect to each row. To achieve a better processing efficiency, golden section search and dynamic programming are utilised to estimate and , respectively. By performing a rotation around , the disparity distribution of each row becomes very compact. This further improves the accuracy of the road model estimation, as demonstrated by the various experimental results in this paper. Finally, the Otsu’s thresholding method is applied to the transformed disparity map and the roads can be accurately segmented at pixel level.
keywords:disparity transformation, roll angle, golden section search, dynamic programming, Otsu’s thresholding.
3-D information is very important for various intelligent transportation systems (ITSs), such as lane detection Fan2016 (), road condition assessment Fan2018c (), free space estimation Badino2007 () and visual simultaneous localisation and mapping (SLAM) Newman2006 (). Computer stereo vision is one of the most commonly used techniques in ITSs for dense 3-D data acquisition. The latter is achieved by estimating the relative positional difference between each pair of corresponding points in the left and right images. This difference is generally known as disparity Fan2018 (); Fan2018a (). Since Labayrade et al. proposed the concept of ‘v-disparity map’ in 2002, a lot of research has been carried out to extract road areas from dense disparity maps Labayrade2002 (). These algorithms generally fit a linear or quadratic road model to the best path in the v-disparity map and then analyse the difference between the actual and fitted disparity values to determine the road pixels Fan2018d (). However, the non-zero roll angle (see Figure 1) usually makes the best path in the v-disparity map ambiguous (see Figure (b)b), which further severely affects the accuracy of the road model estimation Evans2018 (). Therefore, has to be estimated beforehand to address the above issue.
Over the past decade, various technologies, e.g., inertial measurement units (IMU) Ryu2002 (); Ryu2004 (); EricTseng2007 (); Oh2013 () and passive sensing Schlipsing2011 (); Evans2018 (), have been utilised to estimate . The methods based on IMU usually combine the data obtained from different sensors, e.g., GPS, accelerometers and gyroscopes, to provide an accurate estimation of the vehicle state EricTseng2007 (); Oh2013 (). This not only makes the set-up cost unreasonably high but also introduces additional complexity in the estimation procedure Schlipsing2011 (). Furthermore, the bank angle (see Figure 1) is always assumed to be zero in these approaches and only is considered in the estimation process. However, in real-world applications, is not always zero and hence both angles have to be estimated independently although this is not usually a straightforward process Ryu2004 (). In recent years, some passive sensor-based methods Ozgunalp2017 (); Fan2018 (); Schlipsing2011 (); Schlipsing2012 (); Labayrade2003 (); Skulimowski () have been proposed to estimate . Most of them used a single camera to acquire road data and some ideal assumptions, such as constant lane width, were made to ensure these algorithms work well Labayrade2003 (). However, the roll angle usually changes over time in actual cases, and therefore such assumptions are not always valid. In this regard, some authors resorted to 3-D information in order to estimate roll angle more accurately Ozgunalp2017 (); Evans2018 (); Labayrade2003 (); Skulimowski (). In Labayrade2003 () and Skulimowski (), the authors assumed that the road surface is a ground plane and estimated the roll and pitch angles from a v-disparity map. In Ozgunalp2017 (); Fan2018 (); Evans2018 (), a ground plane (where represents the coordinate of a pixel in the disparity map) is fitted to a small patch which is selected from the near field in the disparity map and . However, the above stereo vision-based algorithms are only suitable for a flat road surface which can be assumed to be a ground plane Fan2018 (). Furthermore, selecting a proper patch for plane fitting is always a challenging task because it may contain an obstacle or a pothole which can severely affect the fitting accuracy Evans2018 (). Hence in this paper, we first propose a highly robust roll angle estimation algorithm which is suitable for both flat and non-flat roads. By performing a rotation around , the disparity distribution of each row become very compact and this further improves the accuracy of the road model estimation Evans2018 (). Then, the dense disparity map is transformed using two parameters and , where denotes the fitted disparity value with respect to each row and it can be computed using our previously published algorithm in Ozgunalp2017 (). In the transformed disparity map, the disparity value of the road pixels become very similar, which makes the extraction of road area much easier.
The proposed algorithm is composed of three main steps: estimation, estimation and disparity transformation. In this paper, the vertical road profile is formulated as a parabola . For the stereo rig which is perfectly horizontal to the road surface, the actual disparity values on each row are very similar but they reduce gradually when decreases. Also, the difference between the actual and fitted disparity values, i.e., and , is close to zero. However, when is non-zero, the actual disparity values on each row will change gradually (see Figure (a)a) and the disparity distribution of each row spreads out (see Figure (b)b). This severely affects the accuracy of the road model estimation and thus should be rectified beforehand. Firstly, the disparity map is rotated around different angles. For each rotation, we fit using least squares fitting (LSF) and obtain the corresponding minimum energy (see Figure 3). can then be estimated by finding the local minima of the curve in Figure 3. To further improve the processing efficiency, we use golden section search (GSS) to reduce the search range, and thus the roll angle can be estimated more efficiently. In the second step, we use our previously published algorithm Ozgunalp2017 (); Fan2018d () to estimate , where the best path is extracted by optimising the v-disparity map using dynamic programming (DP). Finally, the original disparity map is transformed according to the values of and , which makes the road area very distinguishable.
2 Algorithm Description
Roll angle is an important parameter in vehicle control systems (VCSs) because it is commonly used to estimate vehicle states and maintain the stability of the VCSs EricTseng2007 (). In this paper, the input dense disparity map is estimated using our previously published algorithm Fan2017 (); Fan2018 (). The parameter vector can be estimated by solving a least squares problem as follows:
For a stereo rig which is ideally horizontal to the road surface, the roll angle of the binocular system is zero. The disparity values for each row are similar while they change gradually in the vertical direction (see Figure (a)a). However, a non-zero roll angle introduced from the set-up installation makes the disparity values change gradually in each row (see Figure (a)a), and therefore the disparity distribution of each row becomes less compact (see Figure (b)b) compared to the case when is zero (see Figure (b)b). This greatly affects the accuracy of the LSF and makes higher than the desired value. Therefore, the main idea of the proposed roll angle estimation algorithm is to rotate at different angles and then find the angle at which the minimum is obtained. An example of the relationship between and is shown in Figure 3.
After the coordinate translation and rotation, the position in the rotated disparity map has the same disparity value as the position in , and the energy function in Eq. 2 can thus be rewritten as Eq. 5.
An arbitrary can be computed from and using Eq. 4 and the corresponding is obtained by solving the energy minimisation problem in Eq. 1 using the LSF. It is to be noted that whether the disparity map is rotated around or , the same will be obtained because and . Therefore, we set the interval of to and find the desirable roll angle which corresponds to the local minima as shown in Figure 3.
However, finding the local minima is a computationally intensive task because we have to go through the whole interval. Furthermore, in order to obtain an accurate which corresponds to the minimum , the step size should be set to a very small and practical value. Hence in this paper, GSS is utilised to reduce the search range within the interval . The procedures of the proposed estimation algorithm are given in algorithm 1, where represents the golden section factor. More details on the GSS are available in Pedregal2006 ().
The dense disparity map is rotated around which is estimated using algorithm 1. The rotated disparity map is illustrated in Figure (a)a. Then, we create the corresponding v-disparity map (see Figure (b)b), where we can observe that the disparity distribution of each row becomes very compact. The performance evaluation of the proposed estimation algorithm will be discussed in Section 3.
The disparity projection of the road pixels on the v-disparity map can be extracted using DP by searching for every possible solution and selecting the one with the minimum energy, where and record the path of the optimal solution. More details on DP are provided in our previous work Ozgunalp2017 (). In this section, we mainly discuss the optimisation of estimation where Random Sample Consensus (RANSAC) is utilised to reduce the effects caused by outliers. The steps of estimating are given in algorithm 2. The RANSAC iterates times, where is empirically set to in this paper. For each iteration, we select a set of candidates from the optimal solution and interpolate them into a parabola. In this way the corresponding parameter vector can be estimated and the number of inliers and outliers can be computed. We record the parameters of and the corresponding ratio for each iteration, where and denote the number of inliers and outliers, respectively. Then, we select the parameters of which correspond to the smallest ratio . Finally, are removed from and the parameters of are updated iteratively by fitting a parabola to the updated . The iterations continue until the number of outliers reaches zero. The fitted disparity for each row can thus be computed as: .
2.3 Disparity Transformation and Road Segmentation
Using the values of and estimated in subsections 2.1 and 2.2, the disparity map in Figure (a)a can now be transformed. Firstly, each disparity value in the rotated disparity map is updated with , where is set to in this paper to ensure that the disparity values in the updated are non-negative. The updated is then rotated around to get the transformed disparity map . More details on the proposed disparity transformation method are given in algorithm 3. The corresponding result is shown in Figure (a)a, where we can clearly see that the disparity value of the road pixels becomes very similar but they differs significantly from those of the pothole pixels. Then, we perform Otsu’s thresholding method on the transformed disparity map and the road area can be easily extracted. The corresponding road segmentation result is shown in Figure (b)b, where the area in violet is the extracted road surface.
3 Experimental Results
We first analyse the computational complexity of the proposed estimation algorithm. The computational complexity for estimating the roll angle is , where is the step size. The GSS reduces the search range exponentially as the interval size becomes only after the th iteration. This reduces the computational complexity to . In our experiments, is set to (approximately ), and the GSS-based estimation algorithm only needs to iterate times to get the desirable roll angle with a precision of . The proposed algorithm is implemented in MATLAB 2018a platform on an Intel Core i7-8700k CPU (3.7GHz) using a single thread. The runtime for a single frame with a resolution of is 1.62 seconds. Although the proposed algorithm does not run in real time, we believe that its speed can be increased in the future by exploiting the parallel computing architectures.
To evaluate the accuracy of the proposed estimation algorithm, we create a ground truth image (resolution: ), as shown in Figure (a)a, to simulate the disparity map of a non-flat road surface. The parabola of the road model is set to: . Then, the disparity map in Figure (a)a is rotated at different angles between and . In each rotation, we estimate a roll angle and compute the error between and its ground truth . The maximum error is less than (around ) and the average of the errors is around (around ). Furthermore, Gaussian white noise is added to the created disparity map (see Figure (b)b), where is a random decimal number between and , and is scale parameter to control the intensity of the noise. In our experiments, is set to to overestimate the noise. An example of the relationship between and is shown in Figure 7, where the average of the errors is approximately (The maximum error is ) and is relatively large when is around . This indicates that our proposed roll angle estimation can ensure high accuracy even when the disparity map is affected by noise.
We further evaluate the algorithm’s accuracy using the stereo sequences from the EISATS database Vaudrey2008 (); Wedel2008 (), where the roll angle is zero and some vehicles are on the road surface. Some examples of the experimental results are given in Figure 8. The disparity value of the road pixels become similar while the obstacle disparities change gradually. This occurs because the fitted disparity value decreases gradually from the bottom of the disparity map to the top. However, when becomes smaller than the horizontal coordinate of the vanishing point, becomes negative as proved in Ozgunalp2017 (). The average of the roll angle errors for the EISATS database is approximately which is still low.
In this paper, a novel disparity transformation algorithm was proposed to extract road areas from dense disparity maps. By transforming the dense disparity map, the disparity value of the road pixels become very similar, while they differ greatly from those of obstacles and potholes. The roll angle of a stereo rig can be estimated accurately by rotating the disparity map at different angles and identifying the angle which corresponds to the minimum fitting energy. To lower the computational complexity of estimation, we utilised the golden section search to reduce the search range within the interval for each iteration. Rotating the disparity map around the estimated roll angle makes the disparity distribution of each row become very compact, which greatly improves the accuracy of the dynamic programming and the least squares fitting. Following the disparity transformation, the road area becomes highly distinguishable and can be extracted much easier.
- (1) R. Fan, V. Prokhorov, N. Dahnoun, Faster-than-real-time linear lane detection implementation using SoC DSP tms320C6678, in: Proc. IEEE Int. Conf. Imaging Systems and Techniques (IST), 2016, pp. 306–311. doi:10.1109/IST.2016.7738242.
- (2) R. Fan, Y. Liu, X. Yang, M. J. Bocus, N. Dahnoun, S. Tancock, Real-time stereo vision for road surface 3-d reconstructionarXiv:http://arxiv.org/abs/1807.07433v1.
- (3) H. Badino, U. Franke, R. Mester, Free space computation using stochastic occupancy grids and dynamic programming, in: Workshop on Dynamical Vision, ICCV, Rio de Janeiro, Brazil, Vol. 20, Citeseer, 2007.
- (4) P. Newman, D. Cole, K. Ho, Outdoor slam using visual appearance and laser ranging, in: Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, IEEE, 2006, pp. 1180–1187.
- (5) R. Fan, X. Ai, N. Dahnoun, Road surface 3D reconstruction based on dense subpixel disparity map estimation, IEEE Transactions on Image Processing PP (99) (2018) 1. doi:10.1109/TIP.2018.2808770.
- (6) R. Fan, Y. Liu, M. J. Bocus, M. Liu, Real-time subpixel fast bilateral stereoarXiv:http://arxiv.org/abs/1807.02044v2.
- (7) R. Labayrade, D. Aubert, J.-P. Tarel, Real time obstacle detection in stereovision on non flat road geometry through” v-disparity” representation, in: Intelligent Vehicle Symposium, 2002. IEEE, Vol. 2, IEEE, 2002, pp. 646–651.
- (8) R. Fan, N. Dahnoun, Real-time stereo vision-based lane detection system, Measurement Science and Technology 29 (7) (2018) 074005.
- (9) M. Evans, R. Fan, N. Dahnoun, Iterative roll angle estimation from dense disparity map, in: 2018 7th Mediterranean Conference on Embedded Computing (MECO), IEEE, 2018.
- (10) J. Ryu, E. J. Rossetter, J. C. Gerdes, Vehicle sideslip and roll parameter estimation using gps, in: Proceedings of the AVEC International Symposium on Advanced Vehicle Control, 2002, pp. 373–380.
- (11) J. Ryu, J. C. Gerdes, Estimation of vehicle roll and road bank angle, in: American Control Conference, 2004. Proceedings of the 2004, Vol. 3, IEEE, 2004, pp. 2110–2115.
- (12) H. Eric Tseng, L. Xu, D. Hrovat, Estimation of land vehicle roll and pitch angles, Vehicle System Dynamics 45 (5) (2007) 433–443.
- (13) J. Oh, S. B. Choi, Vehicle roll and pitch angle estimation using a cost-effective six-dimensional inertial measurement unit, Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 227 (4) (2013) 577–590.
- (14) M. Schlipsing, J. Schepanek, J. Salmen, Video-based roll angle estimation for two-wheeled vehicles, in: Intelligent Vehicles Symposium (IV), 2011 IEEE, IEEE, 2011, pp. 876–881.
- (15) U. Ozgunalp, R. Fan, X. Ai, N. Dahnoun, Multiple lane detection algorithm based on novel dense vanishing point estimation, IEEE Transactions on Intelligent Transportation Systems 18 (3) (2017) 621–632.
- (16) M. Schlipsing, J. Salmen, B. Lattke, K. G. Schröter, H. Winner, Roll angle estimation for motorcycles: Comparing video and inertial sensor approaches, in: Intelligent Vehicles Symposium (IV), 2012 IEEE, IEEE, 2012, pp. 500–505.
- (17) R. Labayrade, D. Aubert, A single framework for vehicle roll, pitch, yaw estimation and obstacles detection by stereovision, in: Intelligent Vehicles Symposium, 2003. Proceedings. IEEE, IEEE, 2003, pp. 31–36.
- (18) P. Skulimowski, M. Owczarek, P. Strumiłło, Ground plane detection in 3d scenes for an arbitrary camera roll rotation through ’v-disparity’ representation.
- (19) R. Fan, N. Dahnoun, Real-time implementation of stereo vision based on optimised normalised cross-correlation and propagated search range on a gpu, in: Imaging Systems and Techniques (IST), 2017 IEEE International Conference on, IEEE, 2017, pp. 1–6.
- (20) P. Pedregal, Introduction to optimization, Vol. 46, Springer Science & Business Media, 2006.
- (21) T. Vaudrey, C. Rabe, R. Klette, J. Milburn, Differences between stereo and motion behaviour on synthetic and real-world stereo sequences, in: Image and Vision Computing New Zealand, 2008. IVCNZ 2008. 23rd International Conference, IEEE, 2008, pp. 1–6.
- (22) A. Wedel, C. Rabe, T. Vaudrey, T. Brox, U. Franke, D. Cremers, Efficient dense scene flow from sparse or dense stereo data, in: European conference on computer vision, Springer, 2008, pp. 739–751.