FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Abstract

The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

\cvprfinalcopy

1 Introduction

The FlowNet by Dosovitskiy \etal [11] represented a paradigm shift in optical flow estimation. The idea of using a simple convolutional CNN architecture to directly learn the concept of optical flow from data was completely disjoint from all the established approaches. However, first implementations of new ideas often have a hard time competing with highly fine-tuned existing methods, and FlowNet was no exception to this rule. It is the successive consolidation that resolves the negative effects and helps us appreciate the benefits of new ways of thinking.

At the same time, it resolves problems with small displacements and noisy artifacts in estimated flow fields. This leads to a dramatic performance improvement on real-world applications such as action recognition and motion segmentation, bringing FlowNet 2.0 to the state-of-the-art level.

Figure 1: We present an extension of FlowNet. FlowNet 2.0 yields smooth flow fields, preserves fine motion details and runs at to fps. The accuracy on this example is four times higher than with the original FlowNet.
Figure 2: Schematic view of complete architecture: To compute large displacement optical flow we combine multiple FlowNets. Braces indicate concatenation of inputs. Brightness Error is the difference between the first image and the second image warped with the previously estimated flow. To optimally deal with small displacements, we introduce smaller strides in the beginning and convolutions between upconvolutions into the FlowNetS architecture. Finally we apply a small fusion network to provide the final estimate.

The way towards FlowNet 2.0 is via several evolutionary, but decisive modifications that are not trivially connected to the observed problems. First, we evaluate the influence of dataset schedules. Interestingly, the more sophisticated training data provided by Mayer \etal [19] leads to inferior results if used in isolation. However, a learning schedule consisting of multiple datasets improves results significantly. In this scope, we also found that the FlowNet version with an explicit correlation layer outperforms the version without such layer. This is in contrast to the results reported in Dosovitskiy \etal [11].

As a second contribution, we introduce a warping operation and show how stacking multiple networks using this operation can significantly improve the results. By varying the depth of the stack and the size of individual components we obtain many network variants with different size and runtime. This allows us to control the trade-off between accuracy and computational resources. We provide networks for the spectrum between fps and fps.

Finally, we focus on small, subpixel motion and real-world data. To this end, we created a special training dataset and a specialized network. We show that the architecture trained with this dataset performs well on small motions typical for real-world videos. To reach optimal performance on arbitrary displacements, we add a network that learns to fuse the former stacked network with the small displacement network in an optimal manner.

The final network outperforms the previous FlowNet by a large margin and performs on par with state-of-the-art methods on the Sintel and KITTI benchmarks. It can estimate small and large displacements with very high level of detail while providing interactive frame rates.

2 Related Work

End-to-end optical flow estimation with convolutional networks was proposed by Dosovitskiy \etalin [11]. Their model, dubbed FlowNet, takes a pair of images as input and outputs the flow field. Following FlowNet, several papers have studied optical flow estimation with CNNs: featuring a 3D convolutional network [31], an unsupervised learning objective [1, 34], carefully designed rotationally invariant architectures [29], or a pyramidal approach based on the coarse-to-fine idea of variational methods [21]. None of these methods significantly outperforms the original FlowNet.

An alternative approach to learning-based optical flow estimation is to use CNNs for matching image patches. Thewlis \etal [30] formulate Deep Matching [32] as a convolutional network and optimize it end-to-end. Gadot & Wolf [13] and Bailer \etal [3] learn image patch descriptors using Siamese network architectures. These methods can reach good accuracy, but require exhaustive matching of patches. Thus, they are restrictively slow for most practical applications. Moreover, patch based approaches lack the possibility to use the larger context of the whole image because they operate on small image patches.

Convolutional networks trained for per-pixel prediction tasks often produce noisy or blurry results. As a remedy, out-of-the-box optimization can be applied to the network predictions as a postprocessing operation, for example, optical flow estimates can be refined with a variational approach [11]. In some cases, this refinement can be approximated by neural networks: Chen & Pock  [10] formulate reaction diffusion model as a CNN and apply it to image denoising, deblocking and superresolution. Recently, it has been shown that similar refinement can be obtained by stacking several convolutional networks on top of each other. This led to improved results in human pose estimation [18, 9] and semantic instance segmentation [23]. In this paper we adapt the idea of stacking multiple networks to optical flow estimation.

Our network architecture includes warping layers that compensate for some already estimated preliminary motion in the second image. The concept of image warping is common to all contemporary variational optical flow methods and goes back to the work of Lucas & Kanade [17]. In Brox \etal [6] it was shown to correspond to a numerical fixed point iteration scheme coupled with a continuation method.

The strategy of training machine learning models on a series of gradually increasing tasks is known as curriculum learning [5]. The idea dates back at least to Elman [12], who showed that both the evolution of tasks and the network architectures can be beneficial in the language processing scenario. In this paper we revisit this idea in the context of computer vision and show how it can lead to dramatic performance improvement on a complex real-world task of optical flow estimation.

3 Dataset Schedules

High quality training data is crucial for the success of supervised training. We investigated the differences in the quality of the estimated optical flow depending on the presented training data. Interestingly, it turned out that not only the kind of data is important but also the order in which it is presented during training.

The original FlowNets [11] were trained on the FlyingChairs dataset (we will call it Chairs). This rather simplistic dataset contains about k image pairs of chairs superimposed on random background images from Flickr. Random affine transformations are applied to chairs and background to obtain the second image and ground truth flow fields. The dataset contains only planar motions.

The FlyingThings3D (Things3D) dataset proposed by Mayer \etal [19] can be seen as a three-dimensional version of the FlyingChairs. The dataset consists of k renderings of random scenes showing 3D models from the ShapeNet dataset [24] moving in front of static 3D backgrounds. In contrast to Chairs, the images show true 3D motion and lighting effects and there is more variety among the object models.

We tested the two network architectures introduced by Dosovitskiy \etal [11]: FlowNetS, which is a straightforward encoder-decoder architecture, and FlowNetC, which includes explicit correlation of feature maps. We trained FlowNetS and FlowNetC on Chairs and Things3D and an equal mixture of samples from both datasets using the different learning rate schedules shown in Figure 3. The basic schedule (k iterations) corresponds to Dosovitskiy \etal [11] except some minor changes1. Apart from this basic schedule , we investigated a longer schedule with M iterations, and a schedule for fine-tuning with smaller learning rates. Results of networks trained on Chairs and Things3D with the different schedules are given in Table 1. The results lead to the following observations:

Figure 3: Learning rate schedules: is similar to the schedule in Dosovitskiy \etal [11]. We investigated another longer version and a fine-tuning schedule .
Architecture Datasets


FlowNetS
Chairs - -
Chairs -
Things3D -
mixed -
Chairs Things3D -


FlowNetC
Chairs - -
Chairs Things3D -
Table 1: Results of training FlowNets with different schedules on different datasets (one network per row). Numbers indicate endpoint errors on Sintel train clean. mixed denotes an equal mixture of Chairs and Things3D. Training on Chairs first and fine-tuning on Things3D yields the best results (the same holds when testing on the KITTI dataset; see supplemental material). FlowNetC performs better than FlowNetS.
Stack Training Warping Warping Loss after EPE on Chairs EPE on Sintel
architecture enabled included gradient test train clean
Net1 Net2 enabled Net1 Net2


Net1
Net1 Net2
Net1 Net2
Net1 Net2
Net1 W Net2 2.93
Net1 W Net2
Net1 W Net2 1.78
Table 2: Evaluation of options when stacking two FlowNetS networks (Net1 and Net2). Net1 was trained with the Chairs Things3D schedule from Section 3. Net2 is initialized randomly and subsequently, Net1 and Net2 together, or only Net2 is trained on Chairs with ; see text for details. When training without warping, the stacked network overfits to the Chairs dataset. The best results on Sintel are obtained when fixing Net1 and training Net2 with warping.

The order of presenting training data with different properties matters. Although Things3D is more realistic, training on Things3D alone leads to worse results than training on Chairs. The best results are consistently achieved when first training on Chairs and only then fine-tuning on Things3D. This schedule also outperforms training on a mixture of Chairs and Things3D. We conjecture that the simpler Chairs dataset helps the network learn the general concept of color matching without developing possibly confusing priors for 3D motion and realistic lighting too early. The result indicates the importance of training data schedules for avoiding shortcuts when learning generic concepts with deep networks.

FlowNetC outperforms FlowNetS. The result we got with FlowNetS and corresponds to the one reported in Dosovitskiy \etal [11]. However, we obtained much better results on FlowNetC. We conclude that Dosovitskiy \etal [11] did not train FlowNetS and FlowNetC under the exact same conditions. When done so, the FlowNetC architecture compares favorably to the FlowNetS architecture.

Improved results. Just by modifying datasets and training schedules, we improved the FlowNetS result reported by Dosovitskiy \etal [11] by and the FlowNetC result by .

In this section, we did not yet use specialized training sets for specialized scenarios. The trained network is rather supposed to be generic and to work well in various scenarios. An additional optional component in dataset schedules is fine-tuning of a generic network to a specific scenario, such as the driving scenario, which we show in Section 6.

4 Stacking Networks

4.1 Stacking Two Networks for Flow Refinement

All state-of-the-art optical flow approaches rely on iterative methods [7, 32, 22, 2]. Can deep networks also benefit from iterative refinement? To answer this, we experiment with stacking multiple FlowNetS and FlowNetC architectures.

The first network in the stack always gets the images and as input. Subsequent networks get , , and the previous flow estimate , where denotes the index of the network in the stack.

To make assessment of the previous error and computing an incremental update easier for the network, we also optionally warp the second image via the flow and bilinear interpolation to . This way, the next network in the stack can focus on the remaining increment between and . When using warping, we additionally provide and the error as input to the next network; see Figure 2. Thanks to bilinear interpolation, the derivatives of the warping operation can be computed (see supplemental material for details). This enables training of stacked networks end-to-end.

Table 2 shows the effect of stacking two networks, the effect of warping, and the effect of end-to-end training. We take the best FlowNetS from Section 3 and add another FlowNetS on top. The second network is initialized randomly and then the stack is trained on Chairs with the schedule . We experimented with two scenarios: keeping the weights of the first network fixed, or updating them together with the weights of the second network. In the latter case, the weights of the first network are fixed for the first 400k iterations to first provide a good initialization of the second network. We report the error on Sintel train clean and on the test set of Chairs. Since the Chairs test set is much more similar to the training data than Sintel, comparing results on both datasets allows us to detect tendencies to over-fitting.

We make the following observations: (1) Just stacking networks without warping improves results on Chairs but decreases performance on Sintel, i.e. the stacked network is over-fitting. (2) With warping included, stacking always improves results. (3) Adding an intermediate loss after Net1 is advantageous when training the stacked network end-to-end. (4) The best results are obtained when keeping the first network fixed and only training the second network after the warping operation.

Clearly, since the stacked network is twice as big as the single network, over-fitting is an issue. The positive effect of flow refinement after warping can counteract this problem, yet the best of both is obtained when the stacked networks are trained one after the other, since this avoids over-fitting while having the benefit of flow refinement.

4.2 Stacking Multiple Diverse Networks

Rather than stacking identical networks, it is possible to stack networks of different type (FlowNetC and FlowNetS). Reducing the size of the individual networks is another valid option. We now investigate different combinations and additionally also vary the network size.

We call the first network the bootstrap network as it differs from the second network by its inputs. The second network could however be repeated an arbitray number of times in a recurrent fashion. We conducted this experiment and found that applying a network with the same weights multiple times and also fine-tuning this recurrent part does not improve results (see supplemental material for details). As also done in [18, 10], we therefore add networks with different weights to the stack. Compared to identical weights, stacking networks with different weights increases the memory footprint, but does not increase the runtime. In this case the top networks are not constrained to a general improvement of their input, but can perform different tasks at different stages and the stack can be trained in smaller pieces by fixing existing networks and adding new networks one-by-one. We do so by using the Chairs Things3D schedule from Section 3 for every new network and the best configuration with warping from Section 4.1. Furthermore, we experiment with different network sizes and alternatively use FlowNetS or FlowNetC as a bootstrapping network. We use FlowNetC only in case of the bootstrap network, as the input to the next network is too diverse to be properly handeled by the Siamese structure of FlowNetC. Smaller size versions of the networks were created by taking only a fraction of the number of channels for every layer in the network. Figure 4 shows the network accuracy and runtime for different network sizes of a single FlowNetS. Factor yields a good trade-off between speed and accuracy when aiming for faster networks.

Figure 4: Accuracy and runtime of FlowNetS depending on the network width. The multiplier 1 corresponds to the width of the original FlowNet architecture. Wider networks do not improve the accuracy. For fast execution times, a factor of is a good choice. Timings are from an Nvidia GTX 1080.
Number of Networks
1 2 3 4


Architecture
s ss sss
Runtime 7ms ms ms
EPE
Architecture S SS
Runtime ms ms
EPE


Architecture
c cs css csss
Runtime ms ms ms ms
EPE
Architecture C CS CSS
Runtime ms ms ms
EPE 2.10
Table 3: Results on Sintel train clean for some variants of stacked FlowNet architectures following the best practices of Section 3 and Section 4.1. Each new network was first trained on Chairs with and then on Things3D with (Chairs Things3D schedule). Forward pass times are from an Nvidia GTX 1080.

Notation: We denote networks trained by the Chairs Things3D schedule from Section 3 starting with FlowNet2. Networks in a stack are trained with this schedule one-by-one. For the stack configuration we append upper- or lower-case letters to indicate the original FlowNet or the thin version with of the channels. E.g: FlowNet2-CSS stands for a network stack consisting of one FlowNetC and two FlowNetS. FlowNet2-css is the same but with fewer channels.

Table 3 shows the performance of different network stacks. Most notably, the final FlowNet2-CSS result improves by over the single network FlowNet2-C from Section 3 and by over the original FlowNetC [11]. Furthermore, two small networks in the beginning always outperform one large network, despite being faster and having fewer weights: FlowNet2-ss (M weights) over FlowNet2-S (M weights), and FlowNet2-cs (M weights) over FlowNet2-C (M weights). Training smaller units step by step proves to be advantageous and enables us to train very deep networks for optical flow. At last, FlowNet2-s provides nearly the same accuracy as the original FlowNet [11], while running at frames per second.

5 Small Displacements

5.1 Datasets

While the original FlowNet [11] performed well on the Sintel benchmark, limitations in real-world applications have become apparent. In particular, the network cannot reliably estimate small motions (see Figure 1). This is counter-intuitive, since small motions are easier for traditional methods, and there is no obvious reason why networks should not reach the same performance in this setting. Thus, we examined the training data and compared it to the UCF101 dataset [26] as one example of real-world data. While Chairs are similar to Sintel, UCF101 is fundamentally different (we refer to our supplemental material for the analysis): Sintel is an action movie and as such contains many fast movements that are difficult for traditional methods, while the displacements we see in the UCF101 dataset are much smaller, mostly smaller than pixel. Thus, we created a dataset in the visual style of Chairs but with very small displacements and a displacement histogram much more like UCF101. We also added cases with a background that is homogeneous or just consists of color gradients. We call this dataset ChairsSDHom and will release it upon publication.

5.2 Small Displacement Network and Fusion

We fine-tuned our FlowNet2-CSS network for smaller displacements by further training the whole network stack on a mixture of Things3D and ChairsSDHom and by applying a non-linearity to the error to downweight large displacements2. We denote this network by FlowNet2-CSS-ft-sd. This increases performance on small displacements and we found that this particular mixture does not sacrifice performance on large displacements. However, in case of subpixel motion, noise still remains a problem and we conjecture that the FlowNet architecture might in general not be perfect for such motion. Therefore, we slightly modified the original FlowNetS architecture and removed the stride in the first layer. We made the beginning of the network deeper by exchanging the and kernels in the beginning with multiple kernels3. Because noise tends to be a problem with small displacements, we add convolutions between the upconvolutions to obtain smoother estimates as in [19]. We denote the resulting architecture by FlowNet2-SD; see Figure 2.

Finally, we created a small network that fuses FlowNet2-CSS-ft-sd and FlowNet2-SD (see Figure 2). The fusion network receives the flows, the flow magnitudes and the errors in brightness after warping as input. It contracts the resolution two times by a factor of and expands again4. Contrary to the original FlowNet architecture it expands to the full resolution. We find that this produces crisp motion boundaries and performs well on small as well as on large displacements. We denote the final network as FlowNet2.

6 Experiments

We compare the best variants of our network to state-of-the-art approaches on public bechmarks. In addition, we provide a comparison on application tasks, such as motion segmentation and action recognition. This allows benchmarking the method on real data.

6.1 Speed and Performance on Public Benchmarks

MPI Sintel (train final)

Average EPE

Runtime (milliseconds per frame)

CPU

GPU

Ours

fps

fps

fps

EpicFlow

DeepFlow

FlowField

LDOF

LDOF (GPU)

PCA-Flow

PCA-Layers

DIS-Fast

FlowNetS

FlowNetC

FN2-s

FN2-ss

FN2-css-ft-sd

FN2-CSS-ft-sd

FlowNet2

Figure 5: Runtime vs. endpoint error comparison to the fastest existing methods with available code. The FlowNet2 family outperforms other methods by a large margin. The behaviour for the KITTI dataset is the same; see supplemental material.
Method Sintel clean Sintel final KITTI 2012 KITTI 2015 Middlebury Runtime

AEE AEE AEE AEE Fl-all Fl-all AEE ms per frame

train test train test train test train train test train test CPU GPU


Accurate

EpicFlow [22]

DeepFlow [32] 0.25

FlowFields [2] 1.86 3.75 3.06 24.43% 0.33

LDOF (CPU) [7]

LDOF (GPU) [27]

PCA-Layers [33]


Fast

EPPM [4] 6.49 0.33

PCA-Flow [33] 4.04 5.18 14.01 39.59% 0.70

DIS-Fast [16]

FlowNetS [11]


FlowNetC [11]



FlowNet 2.0

FlowNet2-s

FlowNet2-ss

FlowNet2-css

FlowNet2-css-ft-sd

FlowNet2-CSS 08.94

FlowNet2-CSS-ft-sd


FlowNet2 2.02 3.96 3.14 0.35 0.52

FlowNet2-ft-sintel () () 5.74 28.20%

FlowNet2-ft-kitti () 1.8 () () 11.48%



Table 4: Performance comparison on public benchmarks. AEE: Average Endpoint Error; Fl-all: Ratio of pixels where flow estimate is wrong by both pixels and %. The best number for each category is highlighted in bold. See text for details. train numbers for these methods use slower but better “improved” option. For these results we report the fine-tuned numbers (FlowNetS-ft and FlowNetC-ft).

We evaluated all methods5 on a system with an Intel Xeon E5 with 2.40GHz and an Nvidia GTX 1080. Where applicable, dataset-specific parameters were used, that yield best performance. Endpoint errors and runtimes are given in Table 4.

Sintel: On Sintel, FlowNet2 consistently outperforms DeepFlow [32] and EpicFlow [22] and is on par with FlowFields. All methods with comparable runtimes have clearly inferior accuracy. We fine-tuned FlowNet2 on a mixture of Sintel clean+final training data (FlowNet2–ft-sintel). On the benchmark, in case of clean data this slightly degraded the result, while on final data FlowNet2–ft-sintel is on par with the currently published state-of-the art method DeepDiscreteFlow [14].

KITTI: On KITTI, the results of FlowNet2-CSS are comparable to EpicFlow [22] and FlowFields [2]. Fine-tuning on small displacement data degrades the result. This is probably due to KITTI containing very large displacements in general. Fine-tuning on a combination of the KITTI2012 and KITTI2015 training sets reduces the error roughly by a factor of (FlowNet2-ft-kitti). Among non-stereo methods we obtain the best EPE on KITTI2012 and the first rank on the KITTI2015 benchmark. This shows how well and elegantly the learning approach can integrate the prior of the driving scenario.

Middlebury: On the Middlebury training set FlowNet2 performs comparable to traditional methods. The results on the Middlebury test set are unexpectedly a lot worse. Still, there is a large improvement compared to FlowNetS [11].

Endpoint error vs. runtime evaluations for Sintel are provided in Figure 4. One can observe that the FlowNet2 family outperforms the best and fastest existing methods by large margins. Depending on the type of application, a FlowNet2 variant between 8 to 140 frames per second can be used.

6.2 Qualitative Results

Figures 6 and 7 show example results on Sintel and on real-world data. While the performance on Sintel is similar to FlowFields [2], we can see that on real world data FlowNet 2.0 clearly has advantages in terms of being robust to homogeneous regions (rows 2 and 5), image and compression artifacts (rows 3 and 4) and it yields smooth flow fields with sharp motion boundaries.

Image Overlay Ground Truth FlowFields [2] PCA-Flow [33] FlowNetS [11] FlowNet2
(ms) (ms) (ms) (ms)
Figure 6: Examples of flow fields from different methods estimated on Sintel. FlowNet2 performs similar to FlowFields and is able to extract fine details, while methods running at comparable speeds perform much worse (PCA-Flow and FlowNetS).
Image Overlay FlowFields [2] DeepFlow [32] LDOF (GPU) [27] PCA-Flow [33] FlowNetS [11] FlowNet2
Figure 7: Examples of flow fields from different methods estimated on real-world data. The top two rows are from the Middlebury data set and the bottom three from UCF101. Note how well FlowNet2 generalizes to real-world data, i.e. it produces smooth flow fields, crisp boundaries and is robust to motion blur and compression artifacts. Given timings of methods differ due to different image resolutions.

6.3 Performance on Motion Segmentation and Action Recognition

To assess performance of FlowNet 2.0 in real-world applications, we compare the performance of action recognition and motion segmentation. For both applications, good optical flow is key. Thus, a good performance on these tasks also serves as an indicator for good optical flow.

For motion segmentation, we rely on the well-established approach of Ochs \etal [20] to compute long term point trajectories. A motion segmentation is obtained from these using the state-of-the-art method from Keuper \etal [15]. The results are shown in Table 5. The original model in Ochs \etal [15] was built on Large Displacement Optical Flow [7]. We included also other popular optical flow methods in the comparison. The old FlowNet [11] was not useful for motion segmentation. In contrast, the FlowNet2 is as reliable as other state-of-the-art methods while being orders of magnitude faster.

Optical flow is also a crucial feature for action recognition. To assess the performance, we trained the temporal stream of the two-stream approach from Simonyan \etal [25] with different optical flow inputs. Table 5 shows that FlowNetS [11] did not provide useful results, while the flow from FlowNet 2.0 yields comparable results to state-of-the art methods.

Motion Seg. Action Recog.
F-Measure Extracted Accuracy
Objects
LDOF-CPU [7]

DeepFlow [32]
81.89%

EpicFlow [22]

FlowFields [2]
30/65


FlowNetS [11]



FlowNet2-css-ft-sd

FlowNet2-CSS-ft-sd
79.64%

FlowNet2
79.92% 32/65

Table 5: Motion segmentation and action recognition using different methods; see text for details. Motion Segmentation: We report results using [20, 15] on the training set of FBMS-59 [28, 20] with a density of pixels. Different densities and error measures are given the supplemental material. “Extracted objects” refers to objects with . FlowNetS is evaluated on 28 out of 29 sequences; on the sequence lion02, the optimization did not converge even after one week. Action Recognition: We report classification accuracies after training the temporal stream of  [25]. We use a stack of 5 optical flow fields as input. Due to long training times only selected methods could be evaluated. To reproduce results from [25], for action recognition we use the OpenCV LDOF implementation. Note the generally large difference for FlowNetS and FlowNet2 and the performance compared to traditional methods.

7 Conclusions

We have presented several improvements to the FlowNet idea that have led to accuracy that is fully on par with state-of-the-art methods while FlowNet 2.0 runs orders of magnitude faster. We have quantified the effect of each contribution and showed that all play an important role. The experiments on motion segmentation and action recognition show that the estimated optical flow with FlowNet 2.0 is reliable on a large variety of scenes and applications. The FlowNet 2.0 family provides networks running at speeds from 8 to 140fps. This further extends the possible range of applications. While the results on Middlebury indicate imperfect performance on subpixel motion, FlowNet 2.0 results highlight very crisp motion boundaries, retrieval of fine structures, and robustness to compression artifacts. Thus, we expect it to become the working horse for all applications that require accurate and fast optical flow computation.

Acknowledgements

We acknowledge funding by the ERC Starting Grant VideoLearn, the DFG Grant BR-3815/7-1, and the EU Horizon2020 project TrimBot2020.

Appendix A Video

Please see the supplementary video for FlowNet2 results on a number of diverse video sequences, a comparison between FlowNet2 and state-of-the-art methods, and an illustration of the speed/accuracy trade-off of the FlowNet 2.0 family of models.

Optical flow color coding.

For optical flow visualization we use the color coding of Butler \etal [8]. The color coding scheme is illustrated in Figure 1. Hue represents the direction of the displacement vector, while the intensity of the color represents its magnitude. White color corresponds to no motion. Because the range of motions is very different in different image sequences, we scale the flow fields before visualization: independently for each image pair shown in figures, and independently for each video fragment in the supplementary video. Scaling is always the same for all methods being compared.

Figure 1: Flow field color coding used in this paper. The displacement of every pixel in this illustration is the vector from the center of the square to this pixel. The central pixel does not move. The value is scaled differently for different images to best visualize the most interesting range.

Appendix B Dataset Schedules: KITTI2015 Results

In Table 1 we show more results of training networks with the original FlowNet schedule  [11] and the new FlowNet2 schedules and . We provide the endpoint error when testing on the KITTI2015 train dataset. Table 1 in the main paper shows the performance of the same networks on Sintel. One can observe that on KITTI2015, as well as on Sintel, training with on the combination of Chairs and Things3D works best (in the paper referred to as Chairs Things3D schedule).

Architecture Datasets


FlowNetS
Chairs - -
Chairs -
Things3D -
mixed -
Chairs Things3D -


FlowNetC
Chairs - -
Chairs Things3D -

Table 1: Results of training FlowNets with different schedules on different datasets (one network per row). Numbers indicate endpoint errors on the KITTI2015 training dataset.

Appendix C Recurrently Stacking Networks with the Same Weights

The bootstrap network differs from the succeeding networks by its task (it needs to predict a flow field from scratch) and inputs (it does not get a previous flow estimate and a warped image). The network after the bootstrap network only refines the previous flow estimate, so it can be applied to its own output recursively. We took the best network from Table 2 of the main paper and applied Net2 recursively multiple times. We then continued training the whole stack with multiple Net2. The difference from our final FlowNet2 architecture is that here the weights are shared between the stacked networks, similar to a standard recurrent network. Results are given in Table 2. In all cases we observe no or negligible improvements compared to the baseline network with a single Net2.

Training of Warping
Net2 gradient EPE
enabled enabled
Net1 + Net2 2.93
Net1 + Net2
Net1 + Net2
Net1 + Net2 2.85
Net1 + Net2
Table 2: Stacked architectures using shared weights. The combination in the first row corresponds to the best results of Table 2 from the paper. Just applying the second network multiple times does not yield improvements. In the two bottom rows we show the results of fine-tuning the stack of the top networks on Chairs for 100k more iterations. This leads to a minor improvement of performance.

Appendix D Small Displacements

d.1 The ChairsSDHom Dataset

Figure 2: Images from the ChairsSDHom (Chairs Small Displacement Homogeneous) dataset.

As an example of real-world data we examine the UCF101 dataset [26]. We compute optical flow using LDOF [11] and compare the flow magnitude distribution to the synthetic datasets we use for training and benchmarking, this is shown in Figure 3. While Chairs are similar to Sintel, UCF101 is fundamentally different and contains much more small displacments.

To create a training dataset similar to UCF101, following [11], we generated our ChairsSDHom (Chairs Small Displacement Homogeneous) dataset by randomly placing and moving chairs in front of randomized background images. However, we also followed Mayer \etal [19] in that our chairs are not flat 2D bitmaps as in [11], but rendered 3D objects. Similar to Mayer \etal, we rendered our data first in a “raw” version to get blend-free flow boundaries and then a second time with antialiasing to obtain the color images. To match the characteristic contents of the UCF101 dataset, we mostly applied small motions. We added scenes with weakly textured background to the dataset, being monochrome or containing a very subtle color gradient. Such monotonous backgrounds are not unusual in natural videos, but almost never appear in Chairs or Things3D. A featureless background can potentially move in any direction (an extreme case of the aperture problem), so we kept these background images fixed to introduce a meaningful prior into the dataset. Example images from the dataset are shown in Figure 2.

fraction of displacement bin

ChairsSDHom

UCF101

FlyingThings3D

FlyingChairs

Sintel



Displacement magnitude (zoom into orange box)
Figure 3: Left: histogram of displacement magnitudes of different datasets. y-axis is logarithmic. Right: zoomed view for very small displacements. The Chairs dataset very closely follows the Sintel dataset, while our ChairsSDHom datasets is close to UCF101. Things3D has few small displacements and for larger displacements also follows Sintel and Chairs. The Things3D histogram appears smoother because it contains more raw pixel data and due to its randomization of 6-DOF camera motion.

d.2 Fine-Tuning FlowNet2-CSS-ft-sd

With the new ChairsSDHom dataset we fine-tuned our FlowNet2-CSS network for smaller displacements (we denote this by FlowNet2-CSS-ft-sd). We experimented with different configurations to avoid sacrificing performance on large displacements. We found the best performance can be achieved by training with mini-batches of samples: from Things3D and from ChairsSDHom. Furthermore, we applied a nonlinearity of to the endpoint error to emphasize the small-magnitude flows.

Name Kernel Str. Ch I/O In Res Out Res Input
conv0 1 Images
conv1 2 conv0
conv1_1 1 conv1
conv2 2 conv1_1
conv2_1 1 conv2
conv3 2 conv2_1
conv3_1 1 conv3
conv4 2 conv3_1
conv4_1 1 conv4
conv5 2 conv4_1
conv5_1 1 conv5
conv6 2 conv5_1
conv6_1 1 conv6
pr6+loss6 1 conv6_1
upconv5 2 conv6_1
rconv5 1 upconv5+pr6+conv5_1
pr5+loss5 1 rconv5
upconv4 2 rconv5
rconv4 1 upconv4+pr5+conv4_1
pr4+loss4 1 rconv4
upconv3 2 rconv4
rconv3 1 upconv3+pr4+conv3_1
pr3+loss3 1 rconv3
upconv2 2 rconv3
rconv2 1 upconv2+pr3+conv2_1
pr2+loss2 1 rconv2
Table 3: The details of the FlowNet2-SD architecture.
Name Kernel Str. Ch I/O In Res Out Res Input
conv0 1 Img1+flows+mags+errs
conv1 2 conv0
conv1_1 1 conv1
conv2 2 conv1_1
conv2_1 1 conv2
pr2+loss2 1 conv2_1
upconv1 2 conv2_1
rconv1 1 upconv1+pr2+conv1_1
pr1+loss1 1 rconv1
upconv0 2 rconv1
rconv0 1 upconv0+pr1+conv0
pr0+loss0 1 rconv0
Table 4: The details of the FlowNet2 fusion network architecture.

d.3 Network Architectures

The architectures of the small displacement network and the fusion network are shown in Tables 3 and 4. The input to the small displacement network is formed by concatenating both RGB images, resulting in input channels. The network is in general similar to FlowNetS. Differences are the smaller strides and smaller kernel sizes in the beginning and the convolutions between the upconvolutions.

The fusion network is trained to merge the flow estimates of two previously trained networks, and this task dictates the input structure. We feed the following data into the network: the first image from the image pair, two estimated flow fields, their magnitudes, and finally the two squared Euclidean photoconsistency errors, that is, per-pixel squared Euclidean distance between the first image and the second image warped with the predicted flow field. This sums up to channels. Note that we do not input the second image directly. All inputs are at full image resolution, flow field estimates from previous networks are upsampled with nearest neighbor upsampling.

Appendix E Evaluation

e.1 Intermediate Results in Stacked Networks

The idea of the stacked network architecture is that the estimated flow field is gradually improved by every network in the stack. This improvement has been quantitatively shown in the paper. Here, we additionally show qualitative examples which clearly highlight this effect. The improvement is especially dramatic for small displacements, as illustrated in Figure 5. The initial prediction of FlowNet2-C is very noisy, but is then significantly refined by the two succeeding networks. The FlowNet2-SD network, specifically trained on small displacements, estimates small displacements well even without additional refinement. Best results are obtained by fusing both estimated flow fields. Figure 6 illustrates this for a large displacement case.

Figure 4: Runtime vs. endpoint error comparison to the fastest existing methods with available code. The FlowNet2 family outperforms other methods by a large margin.
Image Overlay
FlowNet2-C output
FlowNet2-CS output
FlowNet2-CSS output
Fused output
FlowFields
FlowNet2-SD output
Image Overlay
FlowNet2-C output
FlowNet2-CS output
FlowNet2-CSS output
Fused output
FlowFields
FlowNet2-SD output
Image Overlay
FlowNet2-C output
FlowNet2-CS output
FlowNet2-CSS output
Fused output
FlowFields
FlowNet2-SD
Figure 5: Three examples of iterative flow field refinement and fusion for small displacements. The motion is very small (therefore mostly not visible in the image overlays). One can observe that FlowNet2-SD output is smoother than FlowNet2-CSS output. The fusion correctly uses the FlowNet2-SD output in the areas where FlowNet2-CSS produces noise due to small displacements.
Image Overlay
FlowNet2-C output
FlowNet2-CS output
FlowNet2-CSS output
Fused output
Ground Truth
FlowNet2-SD output
Figure 6: Iterative flow field refinement and fusion for large displacements. The large displacements branch correctly estimates the large motions; the stacked networks improve the flow field and make it smoother. The small displacement branch cannot capture the large motions and the fusion network correctly chooses to use the output of the large displacement branch.

e.2 Speed and Performance on KITTI2012

Figure 4 shows runtime vs. endpoint error comparisons of various optical flow estimation methods on two datasets: Sintel (also shown in the main paper) and KITTI2012. In both cases models of the FlowNet 2.0 family offer an excellent speed/accuracy trade-off. Networks fine-tuned on KITTI are not shown. The corresponding points would be below the lower border of the KITTI2012 plot.

e.3 Motion Segmentation

Table 5 shows detailed results on motion segmentation obtained using the algorithms from [20, 15] with flow fields from different methods as input. For FlowNetS the algorithm does not fully converge after one week on the training set. Due to the bad flow estimations of FlowNetS [11], only very short trajectories can be computed (on average about frames), yielding an excessive number of trajectories. Therefore we do not evaluate FlowNetS on the test set. On all metrics, FlowNet2 is at least on par with the best optical flow estimation methods and on the VI (variation of information) metric it is even significantly better.


Method
Training set (29 sequences) Test set (30 sequences)
D P R F VI O D P R F VI O

LDOF (CPU) [7]
0.81% 86.73% 73.08% 79.32% 0.267 31/65 0.87% 87.88% 67.70% 76.48% 0.366 25/69
DeepFlow [32] 0.86% 88.96% 76.56% 82.29% 0.296 33/65 0.89% 88.20% 69.39% 77.67% 0.367 26/69
EpicFlow [22] 0.84% 87.21% 74.53% 80.37% 0.279 30/65 0.90% 85.69% 69.09% 76.50% 0.373 25/69
FlowFields [2] 0.83% 87.19% 74.33% 80.25% 0.282 31/65 0.89% 86.88% 69.74% 77.37% 0.365 27/69
FlowNetS [11] 0.45% 74.84% 45.81% 56.83% 0.604 03/65 0.48% 68.05% 41.73% 51.74% 0.60 03/69
FlowNet2-css-ft-sd 0.78% 88.07% 71.81% 79.12% 0.270 28/65 0.81% 83.76% 65.77% 73.68% 0.394 24/69
FlowNet2-CSS-ft-sd 0.79% 87.57% 73.87% 80.14% 0.255 31/65 0.85% 85.36% 68.81% 76.19% 0.327 26/69
FlowNet2 0.80% 89.63% 73.38% 80.69% 0.238 29/65 0.85% 86.73% 68.77% 76.71% 0.311 26/69


LDOF (CPU) [7]
3.47% 86.79% 73.36% 79.51% 0.270 28/65 3.72% 86.81% 67.96% 76.24% 0.361 25/69
DeepFlow [32] 3.66% 86.69% 74.58% 80.18% 0.303 29/65 3.79% 88.58% 68.46% 77.23% 0.393 27/69
EpicFlow [22] 3.58% 84.47% 73.08% 78.36% 0.289 27/65 3.83% 86.38% 70.31% 77.52% 0.343 27/69
FlowFields [2] 3.55% 87.05% 73.50% 79.70% 0.293 30/65 3.82% 88.04% 68.44% 77.01% 0.397 24/69
FlowNetS [11] 1.93% 76.60% 45.23% 56.87% 0.680 03/62 0–/69
FlowNet2-css-ft-sd 3.38% 85.82% 71.29% 77.88% 0.297 26/65 3.53% 84.24% 65.49% 73.69% 0.369 25/69
FlowNet2-CSS-ft-sd 3.41% 86.54% 73.54% 79.52% 0.279 30/65 3.68% 85.58% 67.81% 75.66% 0.339 27/69
FlowNet2 3.41% 87.42% 73.60% 79.92% 0.249 32/65 3.66% 87.16% 68.51% 76.72% 0.324 26/69
T