RGB-T Object Tracking: Benchmark and Baseline
RGB-Thermal (RGB-T) object tracking receives more and more attention due to the strongly complementary benefits of thermal information to visible data. However, RGB-T research is limited by lacking a comprehensive evaluation platform. In this paper, we propose a large-scale video benchmark dataset for RGB-T tracking. It has three major advantages over existing ones: 1) Its size is sufficiently large for large-scale performance evaluation (total frame number: 234K, maximum frame per sequence: 8K). 2) The alignment between RGB-T sequence pairs is highly accurate, which does not need pre- or post-processing. 3) The occlusion levels are annotated for occlusion-sensitive performance analysis of different tracking algorithms. Moreover, we propose a novel graph-based approach to learn a robust object representation for RGB-T tracking. In particular, the tracked object is represented with a graph with image patches as nodes. This graph including graph structure, node weights and edge weights is dynamically learned in a unified ADMM (alternating direction method of multipliers)-based optimization framework, in which the modality weights are also incorporated for adaptive fusion of multiple source data. Extensive experiments on the large-scale dataset are executed to demonstrate the effectiveness of the proposed tracker against other state-of-the-art tracking methods. We also provide new insights and potential research directions to the field of RGB-T object tracking.
Visual tracking is to estimate states of the target object in subsequent frames, given the initial ground-truth bounding box. It is an active and challenging computer vision task, and has drawn a lot of attentions due to its wide applications, such as video surveillance, self-driving cars, and robotics. Despite of many recent breakthroughs in visual tracking, it still faces many challenging problems especially tracking target objects in various environmental conditions (e.g., low illumination, rain, haze and smog, etc.), which significantly affect the imaging quality of visible spectrum.
Integrating visible and thermal (called RGB-T in this paper) spectrum data has been proven to be effective in boosting tracking performance, and also allows tracking target objects in day and night . Visible and thermal information complement each other and contribute to visual tracking in different aspects. On one hand, thermal infrared camera can capture infrared radiation (0.75-13) emitted by subjects with a temperature above absolute zero. Thus they are insensitive to lighting conditions and have a strong ability to penetrate haze and smog. These kind of sensors, therefore, are more effective in capturing objects than visible spectrum cameras under poor lighting conditions and bad weathers. On the other hand, visible spectrum cameras are more effective in separating two moving subjects, which are crossing or moving side (called thermal crossover [2, 3]). Figure 1 shows some typical scenarios.
There are several video benchmark datasets designed for RGB-T tracking purpose such as OSU-CT , LITIV  and GTOT . A major issue of these datasets is their limited size (see Table I for details). To facilitate this research direction, we contribute a more comprehensive video dataset, called RGBT234 in this paper. RGBT234 includes 234 RGB-T videos, each containing a RGB video and a thermal video. Its total frames reach about 234K and the frame of the biggest video pair reaches 8K. Moreover, we annotate each frame with a minimum bounding box covering the target for both modalities, and also take into full consideration of various environmental challenges, such as raining, night, cold and hot days. To analyze the attribute-based performance of different tracking algorithms, we annotate 12 attributes and implement a dozen of baseline trackers, including deep and non-deep ones (e.g., structured SVM, sparse representation and correlation filter based trackers). In addition, we adopt 5 metrics to evaluate RGB-T trackers.
Compared with other existing ones, RGBT234 has the following major advantages. 1) It includes a large amount of video frames, which enables users to perform large-scale performance evaluations. 2) Its alignment across modalities is highly accurate, and does not require any pre- or post-processing (e.g., stereo matching [6, 7] and color correction ) with the advanced imaging mechanisms. 3) Its occlusion levels, including no, partial and heavy occlusions, are annotated for occlusion-sensitive evaluation of different algorithms. 4) Its videos can be captured by both static and moving cameras while keeping the alignment accurate due to the flexibility of our imaging platform. 5) More recent trackers, more attribute annotations and more evaluation metrics are considered.
With this new benchmark dataset, we investigate how to perform robust object tracking under challenging scenarios by adaptively incorporating the information from RGB-T videos, and propose a novel graph-based algorithm to learn RGB-T object features for online tracking.
Specifically, we partition each target bounding box into a set of non-overlapping patches, which are described with RGB and thermal features. However, background information is inevitably included in the bounding box and likely results in model drifting. Thus, the bounding box cannot represent the target object well. We associate a weight to each patch, which reflects its importance to describe the object. We concatenate all patch features within the target bounding box with their corresponding weights to convey structural information of the object while suppressing the pollution of background. The structured SVM algorithm  is then adopted to perform object tracking.
In general, the patch weights are computed via a semi-supervised learning algorithm, e.g., manifold ranking  and random walk . How to learn an optimal affinity matrix is essential. Most of methods construct a fixed-structure graph (e.g., a 8-neighbor graph where each node is only connected with its 8 neighbor nodes), and thus neglect global intrinsic relationship of patches. Li et al.  propose to learn a dynamic graph with the low-rank and sparse representations as the graph affinity to capture the global subspace structure of patches. Although this kind of affinity is somehow valid, the meaning of which is already not the same as the original definition .
Motivated by above observations, we propose a novel graph model to adaptively employ RGB and thermal data to weight learning. We take patches as graph nodes, and pursue a joint sparse representation [2, 14] with patch feature matrix as input. Note that other constraints on the representation matrix, such as low rank , can also be incorporated in our model. We only employ sparse constraints in this paper for an emphasis on efficient performance. To deal with occasional perturbation or malfunction of individual sources, we assign a weight to each modality to represent the reliability, which allows our method to integrate different spectrum data adaptively. Instead of directly using sparse representations, we learn a more meaningful graph affinity with the assumption that patches are more likely located in the same class (the target object or the background) if their sparse representations have a small distance . Based on the graph affinity, the patch weights are computed when giving some initial weights to patches .
It is worth noting that we jointly solve modality weights, sparse representations, and the graph (including graph structure, edge weights and node weights) in a single optimization framework by designing an efficient ADMM (Alternation Direction Method of Multipliers) algorithm .
# total frames
max frames per sequence
This paper makes the following contributions to RGB-T tracking and related applications.
It contributes a comprehensive RGB-T dataset with 234 fully annotated video sequences for large-scale performance evaluation of different tracking algorithms. A dozen of baseline trackers and 5 evaluation metrics are also included in this benchmark. This benchmark will be open to public 111RGB-T tracking dataset webpage:
It proposes a novel graph-based optimization algorithm to learn RGB-T object feature representations for robust object tracking. In particular, the graph, including graph structure, edge weights and node weights, and the modality weights are dynamically learned in a unified ADMM-based optimization framework.
It carries out extensive experiments on the large-scale benchmark dataset. The evaluation results demonstrate the effectiveness of the proposed approach, and we also provide new insights and potential future research directions for RGB-T object tracking.
This paper provides a more complete understanding of initial results , with more background, insights, analysis, and evaluation. In particular, our benchmark advances the previous work  in several aspects. First, we expand our dataset with more videos by taking more challenges into account. For example, we record some new videos in hot days, which bring new challenges to RGB-T tracking. Second, we select more baseline trackers from recent publications, including deep learning based ones, for fair comparison. Third, we adopt more evaluation metrics used in VOT challenges  for comprehensive evaluations of different tracking algorithms. Finally, we carry out extensive experiments on our dataset to evaluate different RGB and RGB-T trackers which demonstrate the effectiveness of the proposed approach. We also provide new insights to the field of RGB-T object tracking and related applications.
Ii Related Work
According to the relevance to our work, we review related works following three research lines, i.e., RGB-T tracking datasets, RGB-T Object Tracking and RGB Object Tracking.
Ii-a RGB-T Tracking Datasets
There are several popular RGB-T video datasets. OSU-CT dataset  contains 6 RGB-T video sequence pairs recorded from two different locations with only people moving, which is obviously not sufficient to evaluate tracking algorithms. Other two RGB-T datasets are collected by [5, 29]. Most of them suffer from limited size, low diversity, and high bias. Li et al.  construct a large size RGB-T dataset called RGBT210 with 210 RGB-T pairs. They record videos under a moving platform which enriches much diversity of the dataset, and annotate the challenging attributes for comprehensive evaluation. This dataset, however, has two issues: 1) It lacks the videos captured in hot days, and is thus highly biasd to environmental temperature since thermal sensors are sensitive to temperature. 2) Videos of two modalities share same bounding box annotations, which sometimes are unreasonable due to existence of alignment errors.
Recently, more and more large datasets for RGB tracking have been proposed [20, 17, 18, 21, 22, 23]. These datasets bring great benefits to visual tracking. Motivated by the observation, we contribute a larger RGB-T tracking dataset with 234 video pairs in total in this paper. This new dataset includes more challenging RGB-T videos, baseline algorithms, attributes and evaluation metrics. Table I shows the detailed comparison of these datasets.
Ii-B RGB-T Object Tracking
Cvejic et al.  investigate the impact of pixel-level fusion of videos from grayscale-thermal surveillance cameras. Recently, sparse representation has been successfully applied to RGB-T tracking. Wu et al.  concatenate the image patches from RGB and thermal sources, and then sparsely represent each sample in the target template space. Liu et al.  fuse the tracking results using min operation on the sparse representation coefficients calculated on both RGB and thermal modalities. These methods may limit the tracking performance in dealing with occasional perturbation or malfunction of individual sources as available spectrums contribute equally. Li et al. [32, 6] introduce a modality weight for each source to represent the imaging quality, and combine the sparse representation in Bayesian filtering framework to perform object tracking.
Ii-C RGB Object Tracking
The methods for RGB object tracking are vast, and a comprehensive review can be found in [18, 23]. Here, according to the relevance to our work, we only briefly review the RGB trackers from the following three categories.
Ii-C1 Structured SVM based Trackers
Hare et al. first apply the structured output prediction algorithm  to visual tracking to avoid the need for an intermediate classification step . They further integrate several practical considerations to improve tracking accuracy and efficiency . Recently, associating each patch with a weight in the structured SVM framework has been proven to be an effective way for suppressing the background effects in visual tracking [11, 12, 36]. In , a random walk restart algorithm is utilized on the fixed-structure graph with patches as nodes to compute patch weights within target object bounding box. The above fixed-structure graph is constructed via only local cues, neglecting global cues that are important for exploring the intrinsic relationship among patches. In , the low-rank and sparse representation is used to learn a dynamic graph with global considerations. Li et al.  take both advantages of  and  to integrate local and global information in graph learning. Following this research line, we propose a novel graph learning algorithm for RGB-T object tracking.
Ii-C2 End-to-End Deep Learning based Trackers
Benefiting from deep learning techniques, many end-to-end deep trackers have been proposed and achieve excellent performance [37, 38, 39, 40, 41]. Hyeonseob et al.  introduce the idea of multi-domain and train a domain-specific layers online for each video to adapt the target representations. It achieves outstanding performance on several benchmark datasets but has a long latency due to online training strategy. Bertinetto et al. regard object tracking as similarity learning problem, and apply the Siamese network, named SiameFC, to visual tracking . However, SiameFC lacks important online update strategy and cannot capture the change of appearance of objects. To handle this problem, Guo et al.  propose a dynamic Siamese network with a fast general transformation learning model. He et al.  design a twofold Siamese network that consists of a semantic branch and an appearance branch to improve the discrimination power of SiameFC in tracking. Besides, the correlation filter is also embedded into deep neural network , named CFNet, to achieve end-to-end tracking.
Ii-C3 Correlation Filter based Trackers
Correlation filters have obtained great achievement in visual tracking due to its accuracy and computational efficiency. The seminal work, MOSSE, is proposed by Bolme et al. , which achieves hundreds of frames per second and high tracking accuracy. Recently, many advanced methods based on correlation filters are proposed in different aspects. For example, the CSK tracker  uses the kernel trick to improve performance. More and more discriminative features are utilized in tracking for robust representation of the objects, such as HOG , color names  and deep features [46, 47]. In addition, several trackers [48, 49] propose a continuous operation to integrate multiple features in a unified correlation filter framework. To adapt the size change, several adaptive scale processing trackers [50, 51] are investigated. To address the boundary effect, SRDCF  is proposed by introducing spatial regularization into the learning of correlation filters to penalize the filter coefficients near the boundary. In CSR-DCF  tracker, spatial reliability map is constructed to adaptively select the object suitable for tracking.
Iii RGBT234 Tracking Benchmark
This section will introduce the details of the newly created benchmark dataset, called RGBT234 in this paper, including imaging hardware setup, dataset annotation and statistics, baseline RGB and RGB-T trackers, and evaluation metrics.
For large-scale performance evaluation of different RGB-T tracking algorithms, we collect 234 RGB-T videos, each containing a RGB video and a thermal video. In this subsection, we introduce the details of our imaging setup, dataset statistics and annotations.
Iii-A1 Imaging Setup
Our imaging hardware consists of a turnable platform, a thermal infrared image (DLS-H37DM-A) and a CCD camera (SONY EXView HAD CC), as shown in Fig. 2 (a). In particular, these two cameras have same imaging parameters, and their optical axises are aligned as parallel by the collimator, as shown in Fig. 2 (b). The above setup makes the common horizon of these two cameras aligned in pixel level. Fig. 2 (c) and (d) show the details.
Iii-A2 Two-stage Annotation
Different from our earlier work , we re-annotate our dataset more accurately in a two-stage fashion. In the first stage, we annotate each frame with a minimum bounding box covering the targets using more reliable source data. Second, we use the annotations in the first stage to adjust position of a minimum bounding box in less reliable source data. All annotations of two steps are done by a full-time annotator to guarantee consistency, respectively.
In addition, we also annotate the attributes for each video sequence for the attribute-sensitive performance analysis, as shown in Table LABEL:tab:2. We present the attribute distribution over the entire dataset in Table III, and some sample pairs with ground truths and attribute annotations in the project page: https://sites.google.com/view/ahutracking001/. Note that each target object is annotated in every frame even when it is fully occluded.
|NO||No Occlusion - the target is not occluded.|
|PO||Partial Occlusion - the target object is partially occluded.|
|LR||Low Resolution - the resolution in the target region is low.|
|DEF||Deformation - non-rigid object deformation.|
Iii-A3 Advantages over existing Datasets
Table I provides summary of existing tracking datasets, including RGB, Thermal, and RGB-T ones. RGB datasets [17, 18, 19, 20, 21, 22, 23] only provide RGB image sequences, which are sensitive to lighting conditions and could be easily affected by bad weathers, such as rain, smog and fog. These datasets are limited to “good” lighting and environmental conditions. Thermal datasets [24, 25, 26, 27] only include thermal videos, and their weakness are revealed when thermal crossover occurs. RGB-T datasets [4, 5, 6] can well address above issues, and the fusion of these two data enables long-term object tracking in day and night.
Our dataset is classified as a RGB-T dataset as it provides aligned RGB and thermal videos. Compared with other existing ones, our dataset has the following advantages: 1) it includes a large amount of annotated highly-accurate frames (total frames: about 234K, maximum frames per sequence: 8K), which allows trackers to perform large-scale performance evaluation. 2) Due to the superior imaging mechanism, its alignment across modalities is more accurate, and does not require any pre- and post-processing (e.g., stereo matching [6, 7] and color correction ). 3) Its occlusion levels, including no, partial and heavy occlusions, are annotated for occlusion-sensitive evaluation of different algorithms. 4) Since the imaging parameters of RGB and thermal cameras in our platform are the same and their optical axis are parallel, its videos can be captured by both static and moving cameras while keeping the alignment accurate.
Iii-B Baseline Approaches
To identify the importance of thermal data, we select several advanced RGB trackers for evaluations, including ECO , C-COT , CFnet , DSST , SOWP , CSR-DCF , SRDCF  and SAMF . The details of these trackers are presented in Table IV.
There are several RGB-T trackers, including L1-PF , JSR  and CSR . We implement L1-PF and JSR according to their papers for comparison as these trackers do not release their code. In addition, we implement several new RGB-T baselines based on some RGB trackers for comprehensive evaluation, including SOWP +RGBT(SVM-based tracker), MEEM +RGBT (SVM-based tracker), KCF +RGBT (correlation filter based tracker), CSR-DCF +RGBT(correlation filter based tracker), CFnet +RGBT(deep learning-based tracker). In particular, KCF+RGBT, SOWP+RGBT, MEEM+RGBT first concatenate RGB and thermal features into a single vector, and then employ the corresponding RGB trackers to perform object tracking. CSR-DCF+RGBT and CFnet+RGBT are implemented as follows: we first utilize the two-stream corresponding algorithms to track in different modalities data. Then the final tracking results are obtained by averaging tracking results in different modalities. We present the details of these RGB-T trackers in Table IV.
Canny edge detector
Iii-C Evaluation Metrics
There are 5 widely used metrics for tracking performance evaluation [18, 20], i.e., precision rate (PR), success rate (SR), Accuracy, Robustness, and expected average overlap (EAO). Similarly, we employ the following metrics for comprehensive evaluations.
Maximum Precision Rate (MPR). PR is the percentage of frames whose output location is within the given threshold distance of ground truth. That is to say, it computes the average Euclidean distance between the center locations of the tracked target and the manually labeled ground-truth positions of all the frames. Although our alignment between two modalities is highly accurate, there still exist small alignment errors. Therefore, we use maximum precision rate (MPR) instead of PR in this paper. Specifically, for each frame, we compute the above Euclidean distance on both RGB and thermal modalities, and adopt the smaller distance to compute the precision. According to [17, 18], we set the threshold to be 20 pixels to obtain the representative MPR.
Maximum Success Rate (MSR). SR is the ratio of the number of successful frames whose overlap is larger than a threshold. Given a tracked bounding box and the ground-truth bounding box of a target object, the overlap score is defined as , where and represent the intersection and union operators, respectively, and denotes the number of pixels in a region. Similar to MPR, we also define maximum success rate (MSR) to measure the tracker results. By varying the threshold, the MSR plot can be obtained, and we employ the area under curve of MSR plot to define the representative MSR.
Accuracy. Accuracy measures how well the predicted bounding box overlaps with the ground truth bounding box. It calculates the average overlap between the predicted and ground truth bounding boxes during successful tracking periods.
Robustness. Robustness measures how many times the tracker loses the target (fails) during tracking. A failure is indicated when the overlap measurement becomes zero. The per-frame accuracy is obtained as an average over these runs. Averaging per-frame accuracies gives per-sequence accuracy, while per-sequence robustness is computed by averaging failure rates over different runs.
Expected Average Overlap (EAO). Expected average overlap combines the raw values of per-frame accuracies and failures in a principled manner and has a clear practical interpretation . It averages the no-reset overlap of a tracker on several short-term sequences. In other words, it estimates how accurate the estimated bounding box is after a certain number of frames are processed since initialization .
Iv Proposed Approach
Given the tracked bounding box, we partition it into non-overlapping patches, each of which is assigned with a weight to specify its importance to describe the target object. Moreover, we also associate the modality weights to different modalities to reflect their reliabilities. In particular, we propose a graph-based learning algorithm to compute the patch and modality weights together with graph structure optimization. Then, the patch and modality weights and patch features are combined into a single feature descriptor to represent the bounding box. The object tracking is finally performed via Structured SVM algorithm . It should be noted that other tracking-by-detection algorithms can also be used. Fig. 3 shows the overflow of our tracking approach.
Iv-a Patch-based Representation Regularized Graph Learning
In this subsection, we introduce the proposed graph-based learning algorithm that infers the patch weights and the modality weights.
Each bounding box of the target object is partitioned into non-overlapping patches, and a set of low-level appearance features are extracted and further combined into a -dimensional feature vector for characterizing the -th patch in the -th modality. All the feature descriptors of patches in one bounding box form a data matrix , where indicates the index of the modality with the range between 1 and . Herein, we discuss the general case for the scalability, and RGB-T data used in this paper is the special case with .
We take all the above patches as graph nodes and dynamically learn a graph according to the intrinsic relationship of patches instead of using fixed-structure graph in conventional methods [10, 11]. Motivated by the sparse clustering algorithms [56, 57], we assume the foreground or background patches are in the same sparse subspace , and thus each patch descriptor can be sparsely self-represented by a linear combination of remaining patches: , where is a sparse representation coefficient matrix. Sparse constraints can automatically select most informative neighbors for each patch (higher-order relationship), making the graph more powerful and discriminative . Considering the patches are often disturbed by noises and/or corruptions, we introduce a noise matrix to improve the robustness. The joint sparse representation with the convex relaxation  for all modalities can be formulated as:
where and denote the Frobenius norm and the norm of a matrix, respectively. and are the balanced parameters. is the joint sparse representation coefficients matrix, and encourages each patch shares the same pattern across different modalities. denotes the noise matrix, and makes it as the sparse sample-specific corruptions, i.e., some patches are corrupted and others are clean. It is worth noting that other constraints on the representation matrix, such as low rank , can also be incorporated in (1), and we only employ the sparse constraints in this paper with an emphasis on efficient performance.
In (1), different modalities contribute equally, but usually have different imaging qualities in real-life scenarios. Therefore, we assign a weight to each modality to represent the reliability to deal with occasional perturbation or malfunction of individual source, which allows our method to integrate different spectrum data adaptively . We integrate these modality weights in (1), and have
where is the modality weighting vector. is the modality weight in the -th modality. In general, the reconstruction error can measure how well the patch could be sparsely reconstructed from other patches. Therefore, the qualities of different modalities can be reflected by their respective reconstruction errors. From the first term in (2), we can see that our method places larger weights on those modalities which have smaller reconstruction errors, resulting in a quality-aware weight optimization. The last term of (2) is the regularization of , which avoids a degenerate solution of while allowing them to be specified independently. is an adaptive parameter, which is determined after the first iteration .
Most of methods utilize the representation coefficient matrix to define the graph affinity by . Although this kind of affinity is somehow valid, its meaning is already different from the original definition . Recall that we assume the patch descriptor should have a larger probability to be in the same cluster if their representations have a smaller distance. Therefore, instead of directly using sparse representations, we learn a more meaningful graph affinity by the following constraints:
where , are the balanced parameters, and denotes a unit vector. is the desired affinity matrix. reflects the probability of the patch and from the same class based on the distance between their joint sparse representations and across all modalities. The constraints and are to guarantee the probability property of . The last term is to avoid overfitting of .
Given the graph affinity , we can compute the patch node weights in a semi-supervized way. Let be an initial weight vector, in which if is a foreground patch, and is a background patch. is computed by the initial ground-truth (for first frame) or tracking results (for subsequent frames) as follows: for the -th patch, if it belongs to the shrunk region of bounding box and the remaining patches are 0. Fig. 4 shows the details. Similar to the PageRank and spectral clustering algorithm , the patch weights can be calculated as follows:
where and are the balanced parameters. The first term is the smoothness constraint and the second term is the fitting constraint.
In this paper, we aim to jointly optimize the modality weights, the sparse representations, and the graph (including the structure, the edge weights and the node weights) for boosting their respective performance. Therefore, the final formulation of the proposed model can be written by combining (2), (3) and (4):
We design an efficient ADMM (alternating direction method of multipliers) algorithm  to optimize (5), and the efficiency is demonstrated in Fig. 5. Although our model is non-convex, the subproblem of each variable with fixing others is convex and has a closed form solution. Since each subproblem is convex, we can guarantee that the limit point by our algorithm satisfies the Nash equilibrium conditions . The detailed optimization procedure can be referred to our previous work .
Iv-B Tracking via Structured SVM
Iv-B1 Feature Representation
In RGB-T tracking, it is important to construct a robust descriptor of the bounding box . Therefore, we assign a weight to each patch to suppress the background effects, and also assign a weight for each modality to integrate different source data adaptively. Combining these weights with the corresponding patch feature descriptor forms the final feature representation of the bounding box:
From (6), we can see that and play a critical role in the target feature representation.
We adopt the structured SVM (S-SVM)  to perform object tracking in this paper. Note that scale variations emerge frequently during tracking, and thus translation estimation only cannot provide accurate results. In other words, the search space excludes the true state of a target regarding scale. The problem can be alleviated by considering candidates of various sizes. However, this approach may generate too many candidates, increase false positives, and reduce the tracking reliability. Therefore, we decompose the problem of target state estimation into two subproblems: translation estimation and scale estimation [60, 61] as follows.
Iv-B3 Translation estimation
In this work, we combine object representations of multiple modalities with S-SVM to achieve robust RGB-T tracking. Given the bounding box of the target object in previous frame , we first set a searching window in current frame , and sample a set of candidates within the searching window. S-SVM selects the optimal target bounding box in the t-th frame by maximizing a classifier score:
where is a balancing parameter, and is the normal vector of a decision plane of -th frame. Let denote the object descriptor representing a bounding box at the -th frame, and is learned in the initial frame, which can prevent it from learning drastic appearance changes . Instead of using binary-labeled samples, S-SVM employs structured sample that consists of a target bounding box and nearby boxes in the same frame to prevent the labelling ambiguity in training the classifier. Specifically, it constraints that the confidence score of a target bounding box is larger than that of nearby box by a margin determined by the intersection over union overlap ratio (denoted as ) between two boxes:
where , , and is a regularization parameter. By this way, S-SVM can reduce adverse effects of false labelling. To prevent the effects of unreliable tracking results, we update the classifier only when the confidence score of tracking result is larger than a threshold . The confidence score of tracking result in -th frame is defined as the average similarity between the weighted descriptor of the tracked bounding box and the positive support vectors: , where is the set of the positive support vectors at time .
Iv-B4 Scale estimation
During RGB-T tracking, we construct a target pyramid around the estimated translation location for scale estimation . Let be the target size in a test frame and N indicates the number of scales: . For each , we extract an image region of size centered around the estimated location and obtain its classification score . Then, we uniformly resize all image regions with the size , and the optimal scale of target can be estimated by maximizing the classification scores of all resized image regions:
V Performance Evaluation
In this section, we first present the evaluation details, and then show experimental results of 14 tracking algorithms on the RGBT234 dataset with detailed analysis of each challenging factor. At last, the limitations of the proposed approach are analyzed in detail. The source code, the new dataset, the baseline trackers and the evaluation results will be available online for free academic usage and accessible reproducible research.
V-a Parameter Settings
For fair comparison, we fix all parameters and other settings of the proposed approach in all experiments. We partition the bounding box into 64 non-overlapping patches to balance accuracy and efficiency, and extract color (gray for thermal source) and gradient histograms for each patch. The dimension of gradient and each color channel is set to be 8. To improve efficiency, each frame is scaled so that the minimum side length of bounding box is 32 pixels, and the side length of a searching window is fixed to be , where and are the width and height of the scaled bounding box, respectively.
Although (5) seems very complex, its parameters are easy to tune. On one hand, similar to , we simplify the setting as . Following , we set . On the other hand, and control the balance of smoothness and fitness of . According to the setting of similar models [64, 65], we set . In the structured SVM, we empirically set . All parameters are optimized by varying them within a certain scope.
V-B Overall Performance
The overall performance of all the trackers is summarized by the maximum success and precision plots as shown in Fig. 6. Table V presents the evaluation results using other 3 metrics, i.e., Accuracy, Robustness, and Expected Average Overlap (EAO). We analyze these tracking algorithms from the following two lines.
V-B1 Tracking Methods Without Deep Features
The proposed approach (i.e., SGT) performs best over others in MPR and EAO (combination of Accuracy and Robustness), but slightly worse than CSR-DCF +RGBT in MSR (by 1.8%). The lower performance in MSR and Accuracy against others suggests that SGT cannot handle scale variations well, and we will study robust scale handling algorithms in the future. Overall, the results demonstrate the effectiveness of SGT in learning RGB-T object features for visual tracking. Also, we have the following major observations and conclusions. For RGB-T methods, SGT, SOWP +RGBT and CSR-DCF+RGBT rank top 3 and significantly outperform others. Note that SGT and SOWP+RGBT are to learn weighted patch representations and CSR-DCF+RGBT is to integrate segmentation into correlation filter learning, which show the effectiveness of background suppression in RGB-T tracking. Similar observations can also be discovered in the performance comparison of RGB trackers. SGT outperforms SOWP+RGBT suggesting the effectiveness of adaptive fusion of RGB and thermal data. By observing the comparison results between RGB and RGB-T methods, the great performance shows the importance of thermal information in visual tracking.
V-B2 Tracking Methods Based on Deep Features
For comprehensive evaluation, we also select several state-of-the-art RGB trackers using deep features in our platform, including ECO , C-COT , CFnet  and CFnet+RGBT. Benefit from thermal data, SGT outperforms them in MPR and EAO (by 1.8%/0.0139, 0.6%/0.0057, 20.0%/0.0801 and 16.9%/0.0856 in MPR/EAO for ECO, C-COT, CFnet and CFnet+RGBT, respectively). Due to weak scale handling scheme, SGT achieves lower performance than ECO and C-COT in MSR and Accuracy, and achieves lower performance than C-COT in Robustness. Note that SGT has the following advantages over the deep learning based methods. 1) It does not rely on large-scale annotated training samples. We train our model using the ground truth in the first frame, and update it in subsequent frames. 2) It is easy to implement as each subproblem of the proposed model has a closed-form solution.
V-C Attribute-based Performance
To quantify the advantages of different tracking algorithms on RGB234, we present the attribute-based MPR and MSR scores in Table VI, and analyze the details in the following paragraphs.
For partial occlusion (PO), SGT and SOWP+RGBT perform best in MPR and comparable against ECO and C-COT, which suggest that PO introduces some background information that could easily corrupt object detectors or trackers, but the weighted local patch representation is an effective way to alleviate background effects. When heavy occlusion (HO) occurs, their performance decreases much more than ECO and C-COT as the initial seeds in SGT and SOWP+RGBT are unreliable which might result in inaccurate weights in background suppression.
V-C2 Low Illumination
Low illumination (LI) indicates that the imaging quality of visible spectrum is bad, and the thermal information is thus critical for robust tracking. From Table VI, we can see that SGT, SOWP+RGBT and CSR-DCF+RGBT significantly outperform ECO, C-COT, SOWP, CSR-DCF and others. It demonstrates the importance of thermal data in visual tracking.
V-C3 Low Resolution
Although deep learning is a powerful tool in feature representation, it usually performs not well for objects with very low resolution (LR). As shown in Table VI, SGT achieves superior performance over C-COT and ECO in MPR. Therefore, learning non-deep features like in SGT might be a better direction for low-resolution objects tracking.
V-C4 Thermal Crossover
C-COT, ECO and SGT are ranked the top 3 on thermal crossover (TC) and significantly excel the remaining methods. It suggests these trackers could learn a good feature representation to discriminate objects from background in the TC challenge. It is worth noting that SOWP achieves better results than SOWP+RGBT on TC. It suggests that directly fusing RGB and thermal information sometimes is ineffective as some individual source might be noisy data.
The targets with deformation (DEF) usually have irregular shapes and their shapes also vary over time. In such circumstance, the bounding box cannot represent the target well and background suppression is essential for avoiding trackers or detectors corrupted. It is verified by the big superior performance of SGT over other methods and the comparable results of SOWP, SOWP+RGBT and CSR-DCF+RGBT against ECO and C-COT.
V-C6 Fast Motion
The performance on the fast motion (FM) subset depends on the search strategies of trackers. For example, correlation filter-based methods pad original target bounding box with a factor, which is usually small to balance accuracy-efficiency trade-off, while structured SVM-based trackers often set a searching window in a more flexible way. The results verify it, such as the significant performance gain of SOWP+RGBT over CSR-DCF+RGBT and the superior performance of SGT over ECO and C-COT. In addition, the comparable performance of CSR-DCF and CSR-DCF+RGBT, CFnet and CFnet+RGBT also supports the conclusion in TC that the direct fusion strategy sometimes is not a good way for RGB-T tracking.
V-C7 Scale Variation
From the results, we can see that correlation filter- and deep learning-based methods can handle scale variations well due to the discriminative ability of them in slightly different appearances, such as ECO, C-COT, CSR-DCF+RGBT and SRDCF. The performance of CFnet and CFnet+RGBT, SOWP and SOWP+RGBT are comparable, which is similar to TC and FM.
V-C8 Motion Blur
In general, motion blur (MB) results in appearance degradation. And how to learn a good feature is critical to address the MB challenge. Deep learning-based trackers and weighted patch representation-based trackers (like ECO, C-COT, SGT and SOWP-RGBT) excel others in MPR and MSR. Some tracking methods, which regard thermal as an extra channel, may not perform effective integration of different data (for example, CFnet+RGBT is much worse than CFnet).
V-C9 Camera Moving
Camera moving (CM) is also a challenging factor for visual tracking, which usually introduces motion variation and motion blur. From the results, we find SGT can adapt to CM well, followed by C-COT, ECO and SOWP-RGBT. On CM, CFnet+RGBT does not bring much performance gain over CFnet.
V-C10 Background Clutter
For background clutter (BC), SGT and SOWP+RGBT outperform other trackers with a clear margin. It suggests the weighted patch representations and thermal information are beneficial to mitigating background effects in the bounding box descriptions.
In summary, the following major conclusions can be made by the RGBT234 evaluation. 1) Thermal data plays a critical role in visual tracking. 2) SGT could learn a good feature representation via the graph learning algorithm for RGB-T tracking. 3) The weighted patch feature representations and deep learning features are two potential ways for RGB-T tracking performance boosting. 3) Utilizing non-deep features (like in SGT) might be a better direction for low-resolution objects tracking. 4) Direct integration of RGB and thermal information sometimes is ineffective in RGB-T tracking.
Vi Concluding Remark
In this paper, we carry out large scale experiments to evaluate the performance of recent online tracking algorithms, including RGB-, RGB-T- and deep learning-based ones. The results demonstrate that by incorporating depth data, trackers can achieve better performance and handle a variety of challenges much more reliable. We hope that our unified benchmark provides new insights to the field, by making experimental evaluation more standardized and easily accessible.
From the evaluation results, trackers that utilize thermal have advantages especially when the target is small, deformable or under partial occlusion, low illumination, camera moving and background clutter. Target appearance might be disturbed significantly under deformation, occlusion, low illumination and camera moving. And low resolution and background clutter make detection difficult, which are the main causes of model drifting for traditional RGB trackers. However, thermal data can provide complementary information to RGB ones to improve appearance features under deformation, occlusion, camera moving and low resolution, and is also insensitive to lighting conditions and thus performs well in low illumination and background clutter. On the contrary, visible spectrum information can discriminate targets from background when thermal data are unreliable, e.g., thermal crossover occurs.
In addition, based on our evaluation results and observations, we highlight some tracking components which are essential for RGB-T tracking performance improvement. First, adaptive fusion is effective for RGB-T tracking. Modality weights that identify modality qualities can be incorporated into tracking models or approaches to achieve this goal (e.g., SGT). Second, background suppression is critical for effective RGB-T tracking. It can be achieved by weighting local features (e.g., SGT, SOWP+RGBT) and integrating segmentation mask in model learning (e.g., CSR-DCF+RGBT). Third, powerful feature representations are essential for high-performance RGB-T tracking. It can be achieved by incorporating deep learning features (e.g., ECO and C-COT) or weighted features (SGT and SOWP+RGBT) into tracking framework. Finally, components used in conventional RGB trackers are also important for RGB-T tracking, such as motion model especially when the motion of target is large or abrupt. And local models are particularly useful when the appearance of target is partially changed, such as partial occlusion or deformation. Improving above mentioned components will further advance the state-of-the-art of RGB-T object tracking.
-  R. Gade and T. B. Moeslund, “Thermal cameras and applications: a survey,” Machine Vision and Applications, vol. 25, no. 1, pp. 245–262, 2014.
-  H. Liu and F. Sun, “Fusion tracking in color and infrared images using joint sparse representation,” Information Sciences, vol. 55, no. 3, pp. 590–599, 2012.
-  C. Li, X. Wang, L. Zhang, J. Tang, H. Wu, and L. Lin, “Weld: Weighted low-rank decomposition for robust grayscale-thermal foreground detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 4, pp. 725–738, 2016.
-  J. W. Davis and V. Sharma, “Background-subtraction using contour-based fusion of thermal and visible imagery,” Computer Vision and Image Understanding, vol. 106, no. 2, pp. 162–182, 2007.
-  A. Torabi, G. Masse, and G.-A. Bilodeau, “An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications,” Computer Vision and Image Understanding, vol. 116, no. 2, pp. 210–221, 2012.
-  C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, and L. Lin, “Learning collaborative sparse representation for grayscale-thermal tracking,” IEEE Transactions on Image Processing, vol. 25, no. 12, pp. 5743–5756, 2016.
-  S. J. Krotosky and M. M. Trivedi, “On color-, infrared-, and multimodal-stereo approaches to pedestrian detection,” IEEE Trans. Intelligent Transportation Systems, vol. 8, no. 4, pp. 619–629, 2007.
-  S. Hwang, J. Park, N. Kim, and et al., “Multispectral pedestrian detection: Benchmark dataset and baseline,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
-  I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, “Large margin methods for structured and interdependent output variables,” Journal of Machine Learning Research, vol. 6, pp. 1453–1484, 2005.
-  D. Zhou, J. Weston, A. Gretton, O. Bousquet, and B. Scholkopf, “Ranking on data manifolds,” in Proceedings of Neural Information Processing Systems, 2004.
-  H.-U. Kim, D.-Y. Lee, J.-Y. Sim, and C.-S. Kim, “Sowp: Spatially ordered and weighted patch descriptor for visual tracking,” in Proceedings of IEEE International Conference on Computer Vision, 2015.
-  C. Li, L. Lin, W. Zuo, and J. Tang, “Learning patch-based dynamic graph for visual tracking,” in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 4126–4132.
-  X. Guo, “Robust subspace segmentation by simultaneously learning data representations and their affinity matrix,” in Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
-  X. Lan, A. J. Ma, and P. C. Yuen, “Multi-cue visual tracking using robust feature-level fusion based on joint sparse representation,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2014.
-  G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by low-rank representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 171–184, 2013.
-  Z. Lin, R. Liu, and Z. Su, “Linearized alternating direction method with adaptive penalty for low-rank representation,” in Advances in Neural Information Processing Systems 24, 2011.
-  Y. Wu, J. Lim, and M.-H. Yang, “Online object tracking: A benchmark,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2013.
-  ——, “Object tracking benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834–1848, 2015.
-  M. Kristan, R. Pflugfelder, A. Leonardis, J. Matas, and L. C. et al., “The visual object tracking vot2014 challenge results,” in Proceedings in European Conference on Computer Vision, 2014.
-  M. Kristan, J. Matas, A. Leonardis, and M. F. et al., “The visual object tracking vot2015 challenge results,” in ICCV workshop on Visual Object Tracking Challenge, 2015.
-  P. Liang, E. Blasch, and H. Ling, “Encoding color information for visual tracking: Algorithms and benchmark,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5630–5644, 2015.
-  A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah, “Visual tracking: An experimental survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7, pp. 1442–1468, 2014.
-  A. Li, M. Lin, Y. Wu, M. Yang, and S. Yan, “NUS-PRO: A New Visual Tracking Challenge,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 335–349, 2016.
-  J. W. Davis and M. A. Keck, “A two-stage template approach to person detection in thermal imagery,” in Application of Computer Vision, 2005. WACV/MOTIONS ’05 Volume 1. Seventh IEEE Workshops on, 2005.
-  J. Portmann, S. Lynen, M. Chli, and R. Siegwart, “People detection and tracking from aerial thermal views,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2014.
-  Z. Wu, N. Fuller, D. Theriault, and M. Betke, “A thermal infrared video benchmark for visual analysis,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014.
-  M. Felsberg, A. Berg, G. Hager, and e. a. Ahlberg, “The thermal infrared visual object tracking vot-tir2015 challenge results,” in The IEEE International Conference on Computer Vision (ICCV) Workshops, 2015.
-  C. Li, N. Zhao, Y. Lu, C. Zhu, and J. Tang, “Weighted sparse representation regularized graph learning for rgb-t object tracking,” in Proceedings of ACM International Conference on Multimedia, 2017.
-  G.-A. Bilodeau, A. Torabi, and P.-L. S.-C. et al., “Thermal-visible registration of human silhouettes: A similarity measure performance evaluation,” Infrared Physics & Technology, vol. 64, pp. 79–86, 2014.
-  N. Cvejic, S. G. Nikolov, H. D. Knowles, A. Loza, A. Achim, D. R. Bull, and C. N. Canagarajah, “The effect of pixel-level fusion on object tracking in multi-sensor surveillance video,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007.
-  Y. Wu, E. Blasch, G. Chen, L. Bai, and H. Ling, “Multiple source data fusion via sparse representation for robust visual tracking,” in Proceedings of International Conference on Information Fusion, 2011.
-  C. Li, X. Sun, X. Wang, L. Zhang, and J. Tang, “Grayscale-thermal object tracking via multi-task laplacian sparse representation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 47, no. 4, pp. 673–681, 2017.
-  I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, “Large margin methods for structured and interdependent output variables,” J. Mach. Learn. Res., vol. 6, pp. 1453–1484, 2005.
-  S. Hare, A. Saffari, and P. H. S. Torr, “Struck: Structured output tracking with kernels,” in Proceedings of IEEE International Conference on Computer Vision, 2011.
-  S. Hare, S. Golodetz, A. Saffari, V. Vineet, M.-M. Cheng, S. L. Hicks, and P. H. Torr, “Struck: Structured output tracking with kernels,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 10, pp. 2096–2109, 2016.
-  C. Li, X. Wu, Z. Bao, and J. Tang, “Regle: Spatially regularzied graph learning for visual tracking,” in Proceedings of ACM International Conference on Multimedia, 2017.
-  H. Nam and B. Han, “Learning multi-domain convolutional neural networks for visual tracking,” in Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, 2016, pp. 4293–4302.
-  L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in European conference on computer vision, 2016, pp. 850–865.
-  J. Valmadre, L. Bertinetto, J. Henriques, A. Vedaldi, and P. H. Torr, “End-to-end representation learning for correlation filter based tracking,” in Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, 2017, pp. 5000–5008.
-  Q. Guo, W. Feng, C. Zhou, R. Huang, L. Wan, and S. Wang, “Learning dynamic siamese network for visual object tracking,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1–9.
-  A. He, C. Luo, X. Tian, and W. Zeng, “A twofold siamese network for real-time object tracking,” arXiv preprint arXiv:1802.08817, 2018.
-  D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, 2010, pp. 2544–2550.
-  J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the circulant structure of tracking-by-detection with kernels,” in Proceedings of European Conference on Computer Vision, 2012.
-  ——, “High-speed tracking with kernelized correlation filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015.
-  M. Danelljan, F. Shahbaz Khan, M. Felsberg, and J. Van de Weijer, “Adaptive color attributes for real-time visual tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1090–1097.
-  C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang, “Hierarchical convolutional features for visual tracking,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3074–3082.
-  Y. Qi, S. Zhang, L. Qin, H. Yao, Q. Huang, J. Lim, and M.-H. Yang, “Hedged deep tracking,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4303–4311.
-  M. Danelljan, A. Robinson, F. S. Khan, and M. Felsberg, “Beyond correlation filters: Learning continuous convolution operators for visual tracking,” in European Conference on Computer Vision, 2016.
-  M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “Eco: Efficient convolution operators for tracking,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
-  M. Danelljan, G. Hager, F. Khan, and M. Felsberg, “Accurate scale estimation for robust visual tracking,” in Proceedings of British Machine Vision Conference, 2014.
-  Y. Li and J. Zhu, “A scale adaptive kernel correlation filter tracker with feature integration,” in European Conference on Computer Vision, 2014.
-  M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg, “Learning spatially regularized correlation filters for visual tracking,” in Proceedings of the IEEE International Conference on Computer Vision, 2015.
-  A. Lukezic, T. Vojir, L. Cehovin, J. Matas, and M. Kristan, “Discriminative correlation filter with channel and spatial reliability,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
-  J. Zhang, S. Ma, and S. Sclaroff, “MEEM: robust tracking via multiple experts using entropy minimization,” in Proceedings of European Conference on Computer Vision, 2014.
-  M. K. et al., “The visual object tracking vot2016 challenge results,” in Proceedings in European Conference on Computer Vision (ECCV) workshops, 2016.
-  E. Elhamifar and R. Vidal, “Sparse subspace clustering,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2009.
-  S. Yan and H. Wang, “Semi-supervised learning by sparse representation,” in Proceedings of the SIAM International Conference on Data Mining, 2009.
-  A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: Analysis and an algorithm,” in Proceedings of Advances in Neural Information Processing Systems, 2001.
-  Y. Xu and W. Yin, “A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion,” SIAM Journal on Imaging Sciences, vol. 6, no. 3, pp. 1758–1789, 2013.
-  M. Danelljan, G. Häger, F. Khan, and M. Felsberg, “Accurate scale estimation for robust visual tracking,” in Proceedings of British Machine Vision Conference, 2014.
-  C. Ma, X. Yang, C. Zhang, and M.-H. Yang, “Long-term correlation tracking,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015.
-  I. A. Matthews, T. Ishikawa, and S. Baker, “The template update problem,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 810–815, 2004.
-  X. Guo, “Robust subspace segmentation by simultaneously learning data representations and their affinity matrix.” in IJCAI, 2015.
-  C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, “Saliency detection via graph-based manifold ranking,” in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, 2013.
-  C. Li, L. Lin, W. Zuo, and J. Tang, “Learning patch-based dynamic graph for visual tracking.” in AAAI, 2017.