Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation

🔼 This figure illustrates the two-stage Image-to-Video (I2V) framework. Stage 1, Image-to-Motion, takes an input image (x^(0)), its segmentation mask (s^(0)), and a motion-specific text prompt (c_motion) to generate mask-based motion trajectories (s^). Stage 2, Motion-to-Video, uses the input image (x^(0)), the generated trajectories (s^), a global text prompt (c), and object-specific prompts (c_local) to produce the final video output (x^). The framework uses a pre-trained LLM to extract the prompts.

read the caption

Figure 2: Overview of our I2V framework, transforming a reference image x(0)superscript𝑥0x^{(0)}italic_x start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT and text prompt c𝑐citalic_c into a coherent video sequence x^^𝑥\hat{x}over^ start_ARG italic_x end_ARG. A pre-trained LLM is used to derive the motion-specific prompt cm⁢o⁢t⁢i⁢o⁢nsubscript𝑐𝑚𝑜𝑡𝑖𝑜𝑛c_{motion}italic_c start_POSTSUBSCRIPT italic_m italic_o italic_t italic_i italic_o italic_n end_POSTSUBSCRIPT and object-specific prompts cl⁢o⁢c⁢a⁢l={cl⁢o⁢c⁢a⁢l(1),…,cl⁢o⁢c⁢a⁢l(L)}subscript𝑐𝑙𝑜𝑐𝑎𝑙superscriptsubscript𝑐𝑙𝑜𝑐𝑎𝑙1…superscriptsubscript𝑐𝑙𝑜𝑐𝑎𝑙𝐿c_{local}=\{c_{local}^{(1)},\dots,c_{local}^{(L)}\}italic_c start_POSTSUBSCRIPT italic_l italic_o italic_c italic_a italic_l end_POSTSUBSCRIPT = { italic_c start_POSTSUBSCRIPT italic_l italic_o italic_c italic_a italic_l end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( 1 ) end_POSTSUPERSCRIPT , … , italic_c start_POSTSUBSCRIPT italic_l italic_o italic_c italic_a italic_l end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_L ) end_POSTSUPERSCRIPT }, capturing each object’s intended motion. We generate an initial segmentation mask s(0)superscript𝑠0s^{(0)}italic_s start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT from x(0)superscript𝑥0x^{(0)}italic_x start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT using SAM2. In Stage 1, the Image-to-Motion utilizes x(0)superscript𝑥0x^{(0)}italic_x start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT, s(0)superscript𝑠0s^{(0)}italic_s start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT, and cm⁢o⁢t⁢i⁢o⁢nsubscript𝑐𝑚𝑜𝑡𝑖𝑜𝑛c_{motion}italic_c start_POSTSUBSCRIPT italic_m italic_o italic_t italic_i italic_o italic_n end_POSTSUBSCRIPT to generate mask-based motion trajectories s^^𝑠\hat{s}over^ start_ARG italic_s end_ARG that represent object-specific movement paths. In Stage 2, the Motion-to-Video takes as input x(0)superscript𝑥0x^{(0)}italic_x start_POSTSUPERSCRIPT ( 0 ) end_POSTSUPERSCRIPT, the generated trajectories s^^𝑠\hat{s}over^ start_ARG italic_s end_ARG, the text prompt c𝑐citalic_c as a global condition, and object-specific prompts cl⁢o⁢c⁢a⁢lsubscript𝑐𝑙𝑜𝑐𝑎𝑙c_{local}italic_c start_POSTSUBSCRIPT italic_l italic_o italic_c italic_a italic_l end_POSTSUBSCRIPT through a masked attention blocks (Section 3.3), producing the final video x^^𝑥\hat{x}over^ start_ARG italic_x end_ARG.

🔼 Figure 3 illustrates the architecture of the masked attention block, a key component of the proposed Image-to-Video generation model. The figure depicts how the model processes video latent patches (represented as squares, color-coded to identify individual objects within the video frame), and how these patches interact with different types of prompts. Global prompts (gray triangles) provide general information about the scene, while object-specific prompts (color-coded triangles) offer detailed instructions for each object. The illustration highlights the use of four distinct attention mechanisms: standard self-attention (considering all patches), masked self-attention (restricting attention to a single object), standard cross-attention (integrating global prompts), and masked cross-attention (aligning object-specific prompts). This combination of attention mechanisms enables the model to generate accurate and consistent object motion, even in complex, multi-object scenarios.

read the caption

Figure 3: Illustration of the masked attention block. Squares represent video latent patches, color-coded to indicate objects (e.g., cat or dog). Triangles denote prompt tokens: gray for global prompts and object-specific colors for local prompts. The pipeline features self-attention for all patches, masked self-attention restricted to each object, cross-attention integrating global prompts, and masked cross-attention aligning object-specific prompts.

🔼 This figure showcases a qualitative comparison of video generation results. It presents visual examples from the SA-V-128 benchmark dataset, directly comparing videos generated by the proposed ‘Through-The-Mask’ method and a baseline TI2V (Text-to-Image-to-Video) approach. The comparison highlights the differences in the realism and accuracy of the generated videos, particularly in terms of object motion and interaction.

read the caption

Figure 4: Qualitative comparison: Visual examples of generated videos for Through-The-Mask compared to the TI2V baseline on examples from the SA-V-128 benchmark.

🔼 This figure compares the performance of using segmentation masks versus optical flow as intermediate representations for generating videos from images. The top row displays the input image and the text prompt used to guide the animation. The second row shows the segmentation masks generated by the model. These masks highlight the different objects and their respective boundaries in the image. The third row shows the optical flow generated by the model. Optical flow indicates the direction and magnitude of motion between consecutive frames. The fourth row displays the final generated videos using the segmentation masks as the intermediate representation. The fifth row displays the final generated videos using the optical flow as the intermediate representation. By comparing the generated videos from these two different methods, we can understand the strengths and weaknesses of each approach.

read the caption

Figure 5: Qualitative comparison of generated videos using segmentation masks vs optical flow as an intermediate motion representation. The first row shows the input image and text, the second row displays the generated masks, and the third row presents the generated optical flow. The fourth and fifth rows show the generated videos, with the fourth row using our mask-based model and the fifth using our flow-based model.

🔼 This figure shows a qualitative comparison of videos generated using different configurations of the Through-The-Mask model. Four versions are shown: one without any masked attention layers, one with only masked self-attention, one with only masked cross-attention, and one with both (the ‘Ours’ version, representing the full model). The videos all depict a cartoon superhero attempting to perform a prayer pose. The results illustrate how the different attention mechanisms affect the accuracy and smoothness of the superhero’s movements and the overall quality of the generated video. Without masked attention, the pose is incorrect. Self-attention improves smoothness but the pose is still not accurate. Cross-attention yields a correct pose but introduces an artifact (blue fingers). The complete model produces both an accurate and smooth result.

read the caption

Figure 6: Qualitative comparison of generated videos for each configuration of Through-The-Mask. The results highlight differences when applying masked cross-attention (With cross-attn), self-attention (With self-attn), both (Ours), or no masked attention layers (No mask attn). Without masked attention, the cartoon superhero fails to perform a prayer. With masked self-attention, the superhero also fails, but the movement appears smoother and more consistent. With masked cross-attention, the superhero successfully performs the prayer, though his fingers turn blue. When integrating the full masked attention mechanism, the superhero performs the action correctly.

🔼 This figure compares the performance of using segmentation masks versus optical flow as intermediate representations for image-to-video generation. The top row shows the input image and text prompt. The second row displays the segmentation masks generated for each object within the image. The third row shows the optical flow fields that were predicted from the input video. The bottom two rows show the generated videos: the fourth row uses a model that leverages the generated masks, while the fifth row uses a model that leverages the predicted optical flow. The comparison highlights the relative strengths and weaknesses of each intermediate representation in driving the generation process.

read the caption

Figure 7: Qualitative comparison of generated videos using segmentation masks vs optical flow as an intermediate motion representation. The first row shows the input image and text, the second row displays the generated masks, and the third row presents the generated optical flow. The fourth and fifth rows show the generated videos, with the fourth row using our mask-based model and the fifth using our flow-based model.

🔼 This figure shows a qualitative comparison of videos generated using two different methods: Through-The-Mask (TTM) and TI2V. Both methods use the DiT architecture. Each row represents a different video generation task, with the input image and text prompt displayed at the top. The videos generated by TTM are displayed next, followed by the videos generated by the TI2V baseline. This allows for a direct visual comparison of the two methods’ capabilities in generating realistic and coherent videos from a given image and textual description, highlighting Through-The-Mask’s superior performance in generating more accurate and natural-looking results.

read the caption

Figure 8: Qualitative comparison of video generations produced by Through-The-Mask (DiT-based) and the TI2V baseline (DiT-based).

🔼 Table 2 presents a quantitative evaluation of different image-to-video generation models on the Image-Animation-Bench dataset. The metrics used are Fréchet Video Distance (FVD), which measures the visual similarity between generated and ground truth videos; CLIPFrame (CF), which assesses temporal consistency across frames; ViCLIP-T, measuring text-video alignment; ViCLIP-V, evaluating image-video fidelity; and Average Displacement (AD), indicating the level of motion in generated videos. Human evaluation scores, representing the percentage of raters who preferred the results of Through-The-Mask over other methods, are also included for a comprehensive comparison.

read the caption

Table 2: Image-Animation-Bench results. We report FVD, CLIPFrame (CF), ViCLIP-T, ViCLIP-V, and Average Displacement (AD), along with human ratings. Human evaluation shows the percentage of raters that prefer the results of Through-The-Mask.

🔼 This table presents the results of an ablation study conducted on the SA-V-128 benchmark dataset. The study investigates the impact of different attention mechanisms (masked cross-attention, masked self-attention, both, and neither) within both U-Net and DiT-based image-to-video generation models. The table compares various metrics across different attention configurations to assess the contribution of each mechanism to the overall performance of the model.

read the caption

Table 3: Ablation study results on the SA-V-128 benchmark comparing the performance of different attention configurations, in both U-Net and DiT-based models.

🔼 This table presents the results of an ablation study that compares the performance of using segmentation masks versus optical flow as the intermediate motion representation in a two-stage image-to-video generation model. The study evaluates four different model configurations. The first is the baseline model using optical flow. The second and third models utilize segmentation masks, one using only masked cross-attention and the other using both masked cross-attention and masked self-attention. The fourth model is the full THROUGH-THE-MASK model. The metrics compared across the configurations are Fréchet Video Distance (FVD), CLIPFrame (CF), ViCLIP-T, ViCLIP-V, and Average Displacement (AD). Lower FVD indicates higher video quality, while higher values in other metrics usually indicate better performance.

read the caption

Table 4: Ablation study comparing segmentation masks and optical flow as motion trajectory representation. w. Seg refers to models with segmentation-based motion trajectories, while w. OF denotes models with optical flow-based motion trajectories.