Video Motion Transfer with Diffusion Transformers

🔼 DiTFlow uses a reference video to guide the generation of new videos. First, it extracts a patch-wise motion signal (Attention Motion Flow or AMF) from the reference video using a pre-trained Diffusion Transformer (DiT). This AMF is then used to condition the latent representation (z_t) during the video generation process, ensuring the synthesized video mimics the motion of the reference video. The figure highlights the key steps: extracting the AMF from the reference video, using the DiT to generate a new video latent representation, and guiding that latent representation with the AMF to match the reference motion. Additionally, the experiments explored optimizing positional embeddings within the DiT to enable zero-shot motion transfer, where motion is transferred to new videos without additional optimization.

read the caption

Figure 2: Core idea of DiTFlow. We extract the AMF from a reference video and we use that to guide the latent representation ztsubscript𝑧𝑡z_{t}italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT towards the motion of the reference video. In our experiments, we also tested optimizing positional embeddings for improved zero-shot performance.

🔼 Figure 3 illustrates the process of generating Attention Motion Flow (AMF) from a reference video. It begins by processing cross-frame attention maps obtained from a Diffusion Transformer (DiT) model. The argmax operation identifies the strongest attention relationships between patches in consecutive frames, creating a displacement map that shows how patches move over time. For generating new videos, a soft argmax is used instead of argmax to retain gradient information during optimization, ensuring smoother motion transfer. The reference displacement is then used as guidance to constrain the motion of the synthesized video.

read the caption

Figure 3: Guidance. We compute the reference displacement by processing cross-frame attentions with an argmax operation and rearranging them into displacement maps, identifying patch-aware cross-frame relationships. For video synthesis, we do the same operation with a soft argmax to preserve gradients, and impose reconstruction with the reference displacement.

🔼 Figure 4 demonstrates a comparison of motion transfer results between DiTFlow and several baseline methods. The baselines, relying on UNet architectures, struggle with accurate motion transfer because their spatial averaging or localized deviation analysis fails to capture the complete spatiotemporal context of motion. This leads to errors in associating motion to the correct elements within the video frame. In contrast, DiTFlow leverages the attention mechanism within Diffusion Transformers to comprehensively understand the spatiotemporal dynamics of each patch within the video. This results in more precise motion transfer and accurate spatial positioning and sizing of moving elements within the generated video. The figure provides three examples illustrating this difference: a dog running, a bear running, and a parachutist descending. In each example, DiTFlow achieves significantly better motion transfer accuracy than the baselines.

read the caption

Figure 4: Baseline comparison. Baselines associate motion to wrong elements due to poor layout representation typical of UNet-based approaches that do spatial averaging or only consider deviations at each location. DiTFlow captures the spatio-temporal motion of each patch, resulting in correct spatial positioning and sizing of moving elements, e.g. the dog (left), the bear (middle), the parachute (right).

🔼 Figure 5 showcases the qualitative results of DiTFlow, demonstrating its ability to successfully transfer motion from a reference video to newly synthesized videos under diverse conditions. The top row illustrates how altering the text prompt significantly changes the generated video’s appearance (scene, objects, etc.) while preserving the original motion. This highlights DiTFlow’s capacity for content-agnostic motion transfer. The bottom right image further emphasizes this capability, showcasing accurate motion transfer despite substantial changes in the object’s position and scale in the generated video compared to the reference video.

read the caption

Figure 5: Qualitative results of DiTFlow. We are able to perform motion transfer in various conditions. Note how varying the prompt completely changes the scene’s appearance while maintaining consistent motion. We map motion to correct elements even in cases where the motion changes drastically in positioning and size (bottom right).

🔼 This figure displays the results of a human evaluation comparing DiTFlow’s performance to other baselines in video generation. Participants rated generated videos on two aspects using a Likert scale: motion adherence (how well the generated video’s motion matched the reference video) and prompt adherence (how well the generated video matched the textual prompt). The bar chart visually represents the distribution of responses for each model across the levels of agreement (strongly agree, agree, neutral, disagree, strongly disagree). DiTFlow consistently outperforms the baselines in both motion and prompt adherence, demonstrating superior performance in capturing motion from reference videos and adhering to prompts.

read the caption

Figure 6: Human evaluation. We asked humans to evaluate agreement on the quality of generated samples in terms of motion (left) and prompt (right) adherence. DiTFlow consistently outperforms baselines in both evaluations.

🔼 Figure 7a presents a quantitative comparison of the zero-shot motion transfer performance between two versions of the proposed DiTFlow method (optimizing latent representations z_t and positional embeddings p) and the SMM baseline. The results are based on the ‘All’ prompts category (Caption, Subject, Scene) from Table 1 and showcase the effectiveness of the zero-shot approach for various prompts.

read the caption

(a) Quantitative evaluation. SMM and Ours-ztsubscript𝑧𝑡z_{t}italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT follow Table 1 (All)

Reference	Caption			Subject			Scene
Injection
SMM [62]
MOFT [59]
DiTFlow
	“Dog running between poles in an agility course”			“Bear running in a garden”			“Parachuting over a city, aerial view from above”

🔼 This table presents ablation study results focusing on the impact of different choices for the guidance block in the DiTFlow model. It shows the Mean Intersection over Union (MIoU) and Mean Average Precision (mAP) metrics for different configurations, such as using different blocks for optimization, varying the number of denoising steps with guidance, and changing the number of optimization steps. The results help to determine the optimal configuration for the DiTFlow model, balancing performance and computational cost.

read the caption

(a) Guidance block

🔼 This table presents ablation study results on the impact of varying the number of denoising steps on the performance of the DiTFlow model. Specifically, it shows how changing the percentage of denoising steps guided by the AMF (Attention Motion Flow) affects the model’s Motion Fidelity (MF) and Image Quality (IQ) scores. The results are presented for different percentages of guided steps (0%, 20%, 40%, 80%, and 100%), allowing for an analysis of the optimal balance between guidance and computational cost.

read the caption

(b) Denoising steps

🔼 This table shows the ablation study on the number of optimization steps used in DiTFlow. It displays how the performance metrics (MF and IQ) change as the number of optimization steps increases from 1 to 10, allowing one to see the impact of increased computational cost on the quality of the generated video.

read the caption

(c) Optimization steps

🔼 This table presents ablation study results for the DiTFlow model, investigating different aspects of its inference pipeline. The first part (2a) analyzes the contribution of different DiT blocks to motion transfer, showing that earlier blocks are more influential. The second and third parts (2b and 2c) examine the impact of increasing computational resources by varying the number of denoising steps and optimization steps, respectively, demonstrating performance improvements with greater computational power.

read the caption

Table 2: Ablation studies. We investigate our inference setups. In 2(a), we highlight that early blocks in DiTs contribute more to motion. In 2(b) and 2(c), we show that DiTFlow performance can be further boosted by increasing computational power.

🔼 This table presents a subset of the DAVIS dataset used in the paper’s experiments. For each video, three different prompts are provided, categorized as ‘Caption,’ ‘Subject,’ and ‘Scene.’ The ‘Caption’ prompt is a simple description of the video content. The ‘Subject’ prompt alters the subject of the video while maintaining a similar scene. The ‘Scene’ prompt completely changes the scene and context while keeping the original video’s motion as inspiration. This sampling showcases the variety of videos and prompt types employed in the evaluation to thoroughly assess the motion transfer techniques.

read the caption

Table 3: Dataset snippet. Sample of DAVIS videos chosen with associated prompts from each category described in Section 5.