VideoWorld: Exploring Knowledge Learning from Unlabeled Videos

🔼 The figure compares three different approaches to predicting the next move in a game of Go: using only the game state, using raw video, and using video enhanced with latent codes to represent future visual changes. The graph shows that using video with latent dynamics improves learning efficiency. The ‘Action-Value’ metric represents the quality of each move prediction.

read the caption

Figure 2: Comparison of prediction targets. “State”, “Video” and “Video w/ LDM” refer to three different prediction targets: a state sequence (e.g., labeled positions of moves in Go), a raw video sequence, and a video sequence augmented with latent codes representing future visual changes (this approach is adopted by VideoWorld). “Action-Value” denotes the score for each move in the game, with details provided in Sec. 4.2. By combining rich video information with a compact representation of visual changes, VideoWorld enables more effective learning.

🔼 Figure 3 illustrates the architecture of VideoWorld, a novel video generation model designed for knowledge learning from unlabeled videos. The left panel shows the overall architecture, which comprises a VQ-VAE (Vector Quantized Variational Autoencoder) for encoding video frames into discrete tokens, and an autoregressive transformer for predicting the next token (or next frame) based on the previous tokens. The right panel focuses on the Latent Dynamics Model (LDM), a key component of VideoWorld. The LDM efficiently handles long-range dependencies in video sequences by first compressing the visual changes from each frame to its subsequent H frames into a set of latent codes and then seamlessly integrating these codes with the next token prediction paradigm of the autoregressive transformer. This two-stage process enhances both efficiency and effectiveness of the video generation and knowledge acquisition in VideoWorld.

read the caption

Figure 3: Overview of the proposed VideoWorld model architecture. (Left) Overall architecture. (Right) The proposed latent dynamics model (LDM). First, LDM compresses the visual changes from each frame to its subsequent H𝐻Hitalic_H frames into a set of latent codes. Then, an auto-regressive transformer seamlessly integrates the output of LDM with the next token prediction paradigm.

🔼 This figure visualizes the latent codes learned by the Latent Dynamics Model (LDM) during training on Go and robotic manipulation tasks. The left panel shows UMAP projections of latent codes from the Go game, where each point represents a latent code generated by the LDM before quantization. Odd steps correspond to white player moves and even steps to black player moves. Black moves in steps 2, 4, and 6 are highlighted to demonstrate common patterns for new black moves. These patterns are further clarified with additional color and lines on the board. The right panel shows UMAP projections of latent codes from the robotic arm’s movement in the CALVIN dataset. Here, each point represents a latent code, and the points are color-coded according to the magnitude of displacement along the X, Y, and Z axes at intervals of 1, 5, and 10 frames. Purple and red colors indicate the maximum displacement in opposite directions.

read the caption

Figure 4: UMAP projection [34] of the learned latent code on the Go (Left) and CALVIN (right) training set. Each point represents the continuous (pre-quantization) latent code generated by the LDM. In Go examples, odd steps represent white’s moves, and even steps represent black’s moves. We visualize the latent codes of black moves in steps 2/4/6. The legend shows examples of common patterns learned for new black moves. For clarity, these moves are highlighted on the board with added colors and lines to indicate new patterns. On the right, we visualize the latent codes of the robotic arm’s movement along the X/Y/Z axes at intervals of 1, 5, and 10 frames. Points are color-coded by displacement range, with purple and red indicating the maximum displacement in opposite directions along each axis.

🔼 This figure demonstrates a game of Go between the VideoWorld model and KataGO. The VideoWorld model plays as black. The visualization uses UMAP to project the latent codes generated by the model’s Latent Dynamics Model (LDM). The colors of the new stones placed on the Go board correspond to the colors of the projected latent codes, showing how the model’s internal representation (latent codes) relates to its actions in the game. The clustering of the latent codes suggests that the model is not just reacting to the immediate game state but also considering future possibilities, demonstrating forward planning capabilities.

read the caption

Figure 5: Illustration of playing against KataGO and UMAP projection [34] of the predicted latent code. Our model plays as black. The generated latent code is visualized through the LDM decoder and new stones in the visualization are marked with colors to match the legend. The visualization serves as a probe, indicating that the model shows signs of forward planning.

🔼 This figure shows the comparison of prediction targets. It compares three different prediction targets: a state sequence (e.g., labeled positions of moves in Go), a raw video sequence, and a video sequence augmented with latent codes representing future visual changes. The x-axis represents the number of seen samples and the y-axis represents the Action-Value. It demonstrates that VideoWorld, by combining rich video information with a compact representation of visual changes, achieves superior training efficiency compared to using only state or video information.

read the caption

(a)

🔼 This figure shows the results of an ablation study on the compression length of the latent dynamics model (LDM) in the CALVIN environment. The x-axis represents different compression lengths (H), indicating how many future frames are compressed into a latent code. The y-axis shows the task success rate for three robotic manipulation tasks: Push, Open/Close, and Turn On/Off. The baseline represents the performance without LDM. Different compression lengths were tested, revealing the optimal H value for each task that balances compression and information retention for effective learning.

read the caption

(b)

🔼 The figure shows the results of intervening latent codes with different indices. It demonstrates the impact of altering latent codes at different time steps on the model’s performance. By replacing latent codes with random tokens, the experiment shows how altering earlier codes (those representing immediate next steps) has a greater effect than altering later codes. This highlights the importance of the causal relationships and temporal ordering of information within the latent code sequence for effective reasoning and task completion.

read the caption

(c)

🔼 The figure shows the ablation study of different codebook sizes in the latent dynamics model (LDM) on the performance of Go and CALVIN tasks. The results demonstrate how different codebook sizes (729, 15625, 64000, and 262144) impact the model’s ability to learn and achieve high accuracy in both Go and CALVIN tasks, showcasing the importance of selecting an appropriate codebook size for effective knowledge acquisition.

read the caption

(d)

🔼 This ablation study investigates the impact of the data source on the performance of the VideoWorld model. It compares the model’s performance using only human-generated Go data, only KataGo-generated data, and a combination of both. The results demonstrate how different data sources affect the model’s ability to learn and perform the game, highlighting the role of data quality and diversity in knowledge acquisition.

read the caption

(e)

🔼 This figure visualizes the latent codes generated by the Latent Dynamics Model (LDM) during inference for robotic manipulation tasks. The UMAP projection shows how these latent codes (9 codes representing 9 future time steps, H=9) cluster based on the task being performed. Each point in the UMAP plot represents a latent code, and the color indicates the specific task. The images on the right side show the model’s actions. Yellow-background images depict the actual robotic arm movements during inference. Green-background images show the model’s predictions of the next frames while it was training, illustrating its planning ability.

read the caption

Figure 6: Illustration of robotic manipulation and UMAP projection of the predicted latent code during inference. Latent codes are visualized through the LDM decoder. The UMAP projection illustrates the 9 predicted latent codes (i.e. H=9𝐻9H=9italic_H = 9) across different tasks, with each point color-coded by task type. Visualizations with a yellow background show the model’s actual robotic arm control during inference, while those with a green background represent the model’s next-frame predictions during training.

🔼 This figure shows the UMAP projection of the learned latent codes on the Go (left) and CALVIN (right) training sets. Each point represents the continuous (pre-quantization) latent code generated by the LDM. The Go visualizations show odd steps representing white’s moves and even steps representing black’s moves. The legend shows examples of common patterns learned for new black moves; these are highlighted on the board with colors and lines. The CALVIN visualizations show the latent codes of robotic arm movements along the X, Y, and Z axes at different frame intervals. Points are color-coded by displacement range, with purple and red indicating maximum displacement in opposite directions.

read the caption

(a)

🔼 This table presents a comparison of different models’ performance on the CALVIN benchmark, a robotic manipulation task. The benchmark evaluates the ability of models to perform three tasks: Push Blocks, Open/Close Drawer, and Turn On/Off Light. The table compares a baseline model (MCIL), a state-of-the-art model (HULC), a Transformer model trained with labelled data (Oracle), and the VideoWorld model at varying training data sizes. The ‘Input/Output’ column specifies whether the models use video input and labelled action output (Lab), or only video input. The success rate for each task is given as a percentage for each model. The table highlights that VideoWorld, trained solely on unlabeled video data, performs comparably to models trained with labelled data, demonstrating its ability to learn complex tasks from visual data alone.

read the caption

Table 2: Comparison on CALVIN benchmark. “Lab.” means annotated labels. All models have 300M parameters. † and ‡ denote using an additional 10k and 30k CALVIN trajectories for training, respectively.

🔼 This table presents the results of a joint training experiment on two robotic manipulation benchmarks: CALVIN and RLBench. It compares the performance of a VideoWorld model trained only on visual data to an oracle model (Transformer) trained with ground truth action labels. The table shows success rates (percentage of successful task completions) for three distinct tasks across both benchmarks: Push, Open/Close, and Turn on/off in CALVIN; and Close Microwave and Close Fridge in RLBench. The comparison highlights the VideoWorld model’s ability to achieve promising results despite lacking ground truth action labels, demonstrating its capacity for knowledge acquisition from visual inputs alone.

read the caption

Table 3: Results of joint training on CALVIN and RLBench.

🔼 This table presents ablation studies and analysis on the VideoWorld model (50M parameter version). It shows the impact of different design choices on model performance, including varying compression lengths for the latent dynamics model (LDM) in Go and CALVIN environments, experimenting with different codebook sizes in the LDM, and analyzing the effect of different data sources (human vs. KataGo). The results demonstrate the influence of these choices on both action value and accuracy metrics.

read the caption

Table 4: Ablations and analysis. We conduct all experiments based on our 50M model.

🔼 This table presents ablation study results, focusing on the impact of using only latent codes for prediction, without including the video frames. It compares the performance of a 50M parameter model on Go and CALVIN tasks when predicting only latent codes versus predicting both latent codes and video frames, highlighting the contribution of latent codes to model performance. The metrics used are Action-Value and Action Accuracy, demonstrating how the model’s ability to predict and utilize the latent codes translates to overall task performance.

read the caption

Table 5: Latent code prediction only with 50M parameters.

🔼 This table details the hyperparameters used for training both the latent dynamics model (LDM) and the autoregressive transformer. It lists the optimizer, learning rate, weight decay, momentum, batch size, learning rate schedule, warmup iterations, maximum iterations, augmentations, training loss type, and the target for training (reconstruction or next token prediction). Separate configurations are provided for the LDM and the transformer.

read the caption

Table A.1: Training configurations for the latent dynamics model (LDM) and auto-regressive (AR) transformer.