Video Understanding

Edit many objects at once in videos! MIVE does it accurately without affecting other areas, a big step for AI video editing.

InstanceCap improves text-to-video generation through detailed, instance-aware captions.

Gaze-LLE achieves state-of-the-art gaze estimation by using a frozen DINOv2 encoder and a lightweight decoder, simplifying architecture and improving efficiency.

DiTFlow: training-free video motion transfer using Diffusion Transformers, enabling realistic motion control in synthesized videos via Attention Motion Flow.

ObjCtrl-2.5D: Training-free, precise image-to-video object control using 3D trajectories and camera poses.

MobileVD: The first mobile-optimized video diffusion model, achieving 523x efficiency improvement over state-of-the-art with minimal quality loss, enabling realistic video generation on smartphones.

MoViE: Mobile Diffusion for Video Editing achieves 12 FPS video editing on mobile phones by optimizing existing image editing models, achieving a major breakthrough in on-device video processing.

MinT: Generating coherent videos with precisely timed, multiple events via temporal control, surpassing existing methods.

LiFT leverages human feedback, including reasoning, to effectively align text-to-video models with human preferences, significantly improving video quality.

Mimir: A novel framework harmonizes LLMs and video diffusion models for precise text understanding in video generation, producing high-quality videos with superior text comprehension.

VideoGen-of-Thought (VGoT) creates high-quality, multi-shot videos by collaboratively generating scripts, keyframes, and video clips, ensuring narrative consistency and visual coherence.

OmniCreator: Self-supervised unified image+video generation & universal editing.

VideoLights: a novel framework for joint video highlight detection & moment retrieval, boosts performance via feature refinement, cross-modal & cross-task alignment, achieving state-of-the-art results…

PhysGame benchmark unveils video LLMs’ weaknesses in understanding physical commonsense from gameplay videos, prompting the creation of PhysVLM, a knowledge-enhanced model that outperforms existing mo…

Presto: a novel video diffusion model generates 15-second, high-quality videos with unparalleled long-range coherence and rich content, achieved through a segmented cross-attention mechanism and the L…

VISTA synthesizes long-duration, high-resolution video instruction data, creating VISTA-400K and HRVideoBench to significantly boost video LMM performance.

RollingDepth: Achieving state-of-the-art video depth estimation without using complex video models, by cleverly extending a single-image depth estimator.

Trajectory Attention enhances video motion control by injecting trajectory information, improving precision and long-range consistency in video generation.

TeaCache: a training-free method boosts video diffusion model speed by up to 4.41x with minimal quality loss by cleverly caching intermediate outputs.

TAPTRv3 achieves state-of-the-art long-video point tracking by cleverly using spatial and temporal context to enhance feature querying, surpassing previous methods and demonstrating strong performance…