Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

🔼 This figure illustrates how the Opt-CWM model parameterizes counterfactual perturbations to improve motion estimation. Panel (A) shows the architecture, where a learned perturbation generator (δθ) is added to a pre-trained next-frame prediction model (ΨRGB). This generator uses an MLP to predict perturbations based on image content. Panel (B) contrasts hand-designed perturbations, which often fail to move consistently with objects, against learned perturbations, which integrate more seamlessly with the scene dynamics. This shows how the learnable perturbation generator addresses limitations of hand-designed methods. The caption asks how to learn the parameters of the perturbation generator without labeled data, which is answered in Figure 3.

read the caption

Figure 2: Parameterizing the counterfactual intervention policy as an input-conditioned function. (A) Building on a pre-trained RGB-conditioned predictor 𝚿RGBsuperscript𝚿RGB\boldsymbol{\Psi}^{\texttt{RGB}}bold_Ψ start_POSTSUPERSCRIPT RGB end_POSTSUPERSCRIPT, Opt-CWM uses an image-conditioned perturbation prediction function δθsubscript𝛿𝜃\delta_{\theta}italic_δ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT containing a small MLPθ. As illustrated in B, δθsubscript𝛿𝜃\delta_{\theta}italic_δ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT can learn to predict image-conditioned perturbations that blend naturally with the underlying scene, potentially allowing for the perturbation to be accurately carried over to the next frame prediction. But how should the parameters of δθsubscript𝛿𝜃\delta_{\theta}italic_δ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT be learned to achieve this, without any flow supervision labels? See Figure 3.

🔼 Figure 3 illustrates the training process of the Opt-CWM model. Panel A shows the architecture: a pre-trained RGB-conditioned next-frame predictor (ΨRGB) is used as a base model. A learnable counterfactual perturbation generator (FLOWθ) creates perturbations, which are applied to the input frame before feeding it to ΨRGB. The output of ΨRGB is compared with the prediction without perturbation. This difference provides information about motion, which is used to train a flow-conditioned next-frame predictor (Ψflowη). The latter model learns to predict the next frame based on both the original frame and the sparse flow information (which necessitates accurate flow estimates from FLOWθ). Panel B shows how the reconstruction and flow estimation errors change during training. The tight coupling between the two models ensures that improvements in motion estimation directly benefit frame reconstruction, and vice-versa.

read the caption

Figure 3: A generic principle for learning optimal counterfactuals. A) The parameterized counterfactual flow function FLOWθsubscriptFLOW𝜃\textbf{{FLOW}}_{\theta}FLOW start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT extracts motion from a frozen RGB-conditioned predictor 𝚿RGBsuperscript𝚿RGB\boldsymbol{\Psi}^{\texttt{RGB}}bold_Ψ start_POSTSUPERSCRIPT RGB end_POSTSUPERSCRIPT through counterfactual perturbation (details in Figure 2). Its parameters θ𝜃\thetaitalic_θ are trained using gradients from a flow-conditioned predictor 𝚿ηflowsubscriptsuperscript𝚿flow𝜂\boldsymbol{\Psi}^{\texttt{flow}}_{\eta}bold_Ψ start_POSTSUPERSCRIPT flow end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_η end_POSTSUBSCRIPT that is jointly trained to perform next-frame prediction. The predictor 𝚿flowsuperscript𝚿flow\boldsymbol{\Psi}^{\texttt{flow}}bold_Ψ start_POSTSUPERSCRIPT flow end_POSTSUPERSCRIPT can only learn to predict future frames if it is given correct flow vectors. This explicit information bottleneck ensures useful gradients will get passed back to FLOWθsubscriptFLOW𝜃\textbf{{FLOW}}_{\theta}FLOW start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT . This setup allows us to get better extractions from a pre-trained 𝚿RGBsuperscript𝚿RGB\boldsymbol{\Psi}^{\texttt{RGB}}bold_Ψ start_POSTSUPERSCRIPT RGB end_POSTSUPERSCRIPT predictor by training another flow-conditoned predictor 𝚿flowsuperscript𝚿flow\boldsymbol{\Psi}^{\texttt{flow}}bold_Ψ start_POSTSUPERSCRIPT flow end_POSTSUPERSCRIPT using the same principle of next-frame prediction. (B) As a consequence of tight coupling between the flow-conditioned predictor 𝚿flowsuperscript𝚿flow\boldsymbol{\Psi}^{\texttt{flow}}bold_Ψ start_POSTSUPERSCRIPT flow end_POSTSUPERSCRIPT and the learned flow estimation function FLOWθsubscriptFLOW𝜃\textbf{{FLOW}}_{\theta}FLOW start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT, both motion estimation and pixel reconstruction simultaneously improve.

🔼 This figure showcases a qualitative comparison of Opt-CWM’s performance against other state-of-the-art methods on real-world video sequences. The examples highlight scenarios where methods relying on visual similarity or photometric loss fail. Specifically, it demonstrates that baselines struggle with subtle but significant changes within homogeneous scenes, particularly when objects share similar colors and textures (examples (a) through (e)). It also demonstrates the vulnerability of photometric loss-based methods, like SMURF, to variations in light intensity between frames (examples (f) through (h)). In contrast, Opt-CWM, due to its holistic approach which considers scene transformations and dynamics, consistently achieves superior performance in these challenging situations.

read the caption

Figure 4: Qualitative comparison with baselines on real-world videos. The above examples show the failure modes of previous methods that rely on visual similarity or photometric loss. We observe that the baseline models struggle against subtle but functionally important changes in largely homogeneous scenes depicting objects of similar color and texture ((a) - (e)). Further, the use of photometric loss in self-supervised methods such as SMURF can also be susceptible to differences in light intensity across frame pairs ((f) - (h)). Opt-CWM, however, relies on a holistic understanding of scene transformations and object dynamics and is able to find correspondence without arbitrary heuristics.

🔼 This figure visualizes learned perturbation maps generated by a learned MLP (Multilayer Perceptron). The maps show the amplitudes and standard deviations of optimal Gaussian perturbations predicted for each pixel location and color channel across two example frame pairs. The size and magnitude of these perturbations dynamically adapt to local scene content. Specifically, they reflect the presence and characteristics of foreground objects. The perturbations are not uniform but are shaped to match features of the image, suggesting the model has learned to represent scene properties within them.

read the caption

Figure 5: Perturbation maps emergently reflect scene properties. For two example frame pairs, we show the amplitudes and standard deviations, at each spatial position and for each color channel, of the optimal Gaussian perturbations predicted by MLPθ. These “perturbation maps” emergently reflect scene properties, with perturbation parameters varying in size and magnitude depending on where they are located in the image, corresponding to the presence of foreground objects and their parts.

🔼 This figure analyzes the impact of multiscale refinement on the accuracy of flow estimation. The x-axis represents different pixel error thresholds, while the y-axis shows the fraction of points whose error is below each threshold. Different colored bars represent varying numbers of multiscale iterations. The results indicate that 4 zoom iterations provide the best performance overall, particularly when dealing with challenging cases (as demonstrated by its better performance at higher error thresholds).

read the caption

Figure 6: <𝜹avgabsentsubscript𝜹avg<\boldsymbol{\delta}_{\text{avg}}< bold_italic_δ start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT broken down across thresholds (x𝑥xitalic_x-axis). Fraction of points with error less than a fixed threshold, as a function of number of multiscale (MS) iterations, for pixel thresholds 1, 2, 4, 8, and 16. We find that 4 zoom iterations tends to perform the best, especially for robustness on difficult examples (evidenced by better performance on higher thresholds).

🔼 This figure compares the performance of various optical flow estimation methods (Opt-CWM, Doduo, SEA-RAFT, and SMURF) as the gap between frames increases. The x-axis represents the frame gap, while the y-axis shows the average pixel distance (AD) error, a key metric of accuracy. The results reveal that Opt-CWM and Doduo maintain relatively high accuracy even with larger frame gaps, unlike SEA-RAFT and SMURF, whose accuracy significantly decreases as the frame gap and motion magnitude increase. This suggests that Opt-CWM and Doduo are more robust to challenging real-world scenarios with substantial motion between frames.

read the caption

Figure 7: Model comparison as a function of frame gap. Higher frame gaps present harder flow estimation problems due to including more motion, as reflected by improved performance across models in lower frame gap settings. Opt-CWM and Doduo perform better as frame gap increases, in contrast to optical flow methods SEA-RAFT and SMURF which decay in performance as motion magnitude increases, especially on the AD metric.

🔼 Figure 8 presents a detailed analysis of the accuracy of different models in predicting the location of points across various error thresholds. The x-axis represents the error threshold (in pixels), while the y-axis shows the percentage of points for which the predicted location falls within that threshold of the ground truth. The graph compares Opt-CWM with three baseline models: SEA-RAFT, Doduo, and SMURF. Importantly, the evaluation considers videos with large gaps between frames (the TAP-Vid First protocol), making accurate point tracking particularly challenging. The results demonstrate that Opt-CWM consistently achieves higher accuracy across all error thresholds, particularly for larger thresholds. This highlights the robustness of Opt-CWM in handling scenarios with significant motion and longer frame intervals, which cause challenges for other methods.

read the caption

Figure 8: TAP-Vid First: comparing baseline models on <δavgabsentsubscript𝛿avg<\boldsymbol{\delta}_{\text{avg}}< bold_italic_δ start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT broken down across thresholds (x𝑥xitalic_x-axis). Fraction of points with error less than a fixed threshold, as a function of baseline model. Compared to baseline models, Opt-CWM maintains high performance on all thresholds even when making predictions across large frame gaps, as is necessary for TAP-Vid First.

🔼 Figure 9 presents a comparative analysis of various models’ performance on the TAP-Vid CFG (Constant Frame Gap) benchmark, focusing on the average point tracking error. The x-axis represents different error thresholds (in pixels), and the y-axis shows the percentage of points whose error falls below each threshold. The dataset uses a fixed frame gap, making it advantageous for optical flow methods. While supervised and unsupervised optical flow methods perform well at very low error thresholds (under 2 pixels), Opt-CWM demonstrates superior performance overall, outperforming self-supervised approaches and achieving comparable results to the supervised SEA-RAFT method in predicting a higher proportion of points within an acceptable error range.

read the caption

Figure 9: TAP-Vid CFG: comparing baseline models on <δavgabsentsubscript𝛿avg<\boldsymbol{\delta}_{\text{avg}}< bold_italic_δ start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT broken down across thresholds (x𝑥xitalic_x-axis). Fraction of points with error less than a fixed threshold, as a function of baseline model. For fair comparison, we also evaluate on a constant frame gap setting that is more favorable to optical flow baselines. While baseline methods show strong performance for very low thresholds (<2absent2<2< 2 pixels), we see that in general Opt-CWM outperforms self-supervised methods and is comparable with SEA-RAFT in predicting more points within a reasonable boundary.

🔼 This figure visualizes the evolution of learned perturbations throughout the training process of the Opt-CWM model. It shows how the model’s ability to generate effective perturbations improves over time. Initially, the perturbations are scattered and lack coherence. As training progresses, these perturbations become more concentrated and localized, improving the accuracy of the resulting flow predictions. The figure demonstrates this by showing example perturbation maps at various training epochs alongside a comparison of the ground truth flow (green) from the TAP-Vid dataset and the Opt-CWM model’s predicted flow (blue) for those same examples. The decreasing error between the ground truth flow and the model’s prediction, as demonstrated by the diminishing numerical error value shown, is directly correlated to the evolution of the perturbation.

read the caption

Figure 10: Evolution of perturbations across training epochs: We observe how the predicted perturbations change as the model trains. The perturbation starts as a disjoint streak of colors and converges to a localized peak. This in turn increasingly concentrates the difference image 𝚫𝚫\boldsymbol{\Delta}bold_Δ and leads to better flow prediction. Green is the ground truth flow obtained from the TAP-Vid dataset, and blue is our model’s prediction.

🔼 This table presents an ablation study on the TAP-Vid DAVIS First protocol, evaluating the impact of various design choices on the performance of the Opt-CWM model. The study compares different types of perturbations (hand-designed vs. learned) and techniques such as multi-mask inference and multi-scale refinement. The results, measured across multiple metrics, demonstrate that learned, in-distribution perturbations, coupled with multi-mask inference and multi-scale refinement significantly improve the model’s ability to estimate motion.

read the caption

Table 2: Ablation analysis TAP-Vid DAVIS First protocol. We evaluate multi-mask (MM) and multiscale (MS), in addition to comparing our optimized perturbations (“learned”) with the fixed ones (“red square”/“green square”) [4, 42]. MM and MS columns indicate the number of masking or zooming iterations. We observe a clear improvement on all metrics, highlighting the need for bespoke, in-distribution counterfactual perturbations, multi-mask inference and multi-scale refinement.

🔼 This table presents the results of distilling the Opt-CWM model into a smaller, more efficient SEA-RAFT architecture. Instead of using labeled data for training, the researchers used sparsely pseudo-labeled Kinetics data generated by Opt-CWM. The goal was to achieve fast test-time inference while maintaining performance. The results show that this distilled model outperforms the self-supervised SMURF model and is competitive with supervised RAFT models, demonstrating the effectiveness of this approach.

read the caption

Table 3: Opt-CWM Distillation Results. To obtain fast test-time inference with a small model, we distill Opt-CWM into the smaller and more efficient SEA-RAFT architecture by sparsely pseudo-labeling Kinetics with Opt-CWM. This approach outpeforms the self-supervised SMURF and is competitive with the supervised RAFT models, while requiring no labeled training data.

🔼 This table details the hyperparameters used for pre-training the Counterfactual World Model (CWM) which is a core component of the Opt-CWM model. It shows the optimizer used, learning rate, weight decay, momentum, batch size, learning rate schedule, number of warmup and total training epochs, and data augmentation strategies employed.

read the caption

Table 4: Default pre-training setting of CWM

🔼 This table presents an ablation study on the impact of different training-time masking policies for the RGB-conditioned next frame predictor (ΨRGB). It compares the performance of various masking strategies, including those similar to Video-MAE, where a percentage of patches are masked in each frame (55-55 means 55% in frame 1 and 55% in frame 2). It also explores ’tube’ masking (masking the same spatial location across frames) and completely random masking. The results are evaluated using the TAP-Vid DAVIS First protocol with 256x256 resolution, 3 multi-mask iterations, and 2 multiscale iterations. The goal is to show the impact of the masking policy on the final motion estimation performance, demonstrating that the temporally factored masking policy used in the main experiments is superior.

read the caption

Table 5: Ablation analysis of training-time masking policy on TAP-Vid DAVIS First. We train 𝚿RGBsuperscript𝚿RGB\boldsymbol{\Psi}^{\texttt{RGB}}bold_Ψ start_POSTSUPERSCRIPT RGB end_POSTSUPERSCRIPT with different non-temporally factored masking policies more similar to Video-MAE [39, 45]. The notation of 55-55 indicates 55% of patches are masked in the first frame and 55% are masked in the second frame. Tube masking selects patches at the same spatial location over time, whereas random independently samples patches in each frame. MAE-style masking during training is strictly worse than the temporally-factored masking policy we use as the standard in this paper (shown for reference in the bottom row). All experiments here use 256x256 resolution, MM-3 and MS-2.

🔼 This table presents an ablation study on the test-time masking policy used in the Opt-CWM model. Specifically, it investigates the impact of varying the percentage of masked patches in the second frame (the masking ratio) during the inference phase of the model, while keeping other parameters (such as multi-masking iterations and multi-scale refinement) constant at MM=3 and MS=2. The table shows the performance of the model (measured by Average Jaccard (AJ), Average Distance (AD), average percentage of points predicted correctly within various distance thresholds (< avg), Occlusion Accuracy (OA), and Occlusion F1-score (OF1)) across different masking ratios. A reference masking ratio (90%) used in the main experiments is also included for comparison. The experiment was performed on the TAP-Vid DAVIS First benchmark using a 512-resolution RGB-conditioned next frame predictor.

read the caption

Table 6: Ablation analysis of test-time masking policy on TAP-Vid DAVIS First. We evaluate a 512 resolution 𝚿RGBsuperscript𝚿RGB\boldsymbol{\Psi}^{\texttt{RGB}}bold_Ψ start_POSTSUPERSCRIPT RGB end_POSTSUPERSCRIPT across various masking ratios for the second frame using the MM-3 and MS-2 setting. The standard masking ratio for all results in this work is included as 90% (Ref.) in this table.