VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness

🔼 This figure displays a radar chart visualizing the performance of four state-of-the-art (SOTA) video generation models across 18 distinct dimensions defined by the VBench-2.0 benchmark. Each dimension represents a specific aspect of video generation quality, such as human fidelity, controllability, creativity, physics, and common sense. The radar chart allows for a quick visual comparison of the models’ strengths and weaknesses in each of these areas. The scores are normalized, ensuring a fair comparison despite variations in absolute values between the different dimensions. More detailed numerical results are available in Table 2 of the paper.

read the caption

Figure 2: VBench-2.0 Evaluation Results of SOTA Models. The figure presents the evaluation results of four recent state-of-the-art video generation models across 18 VBench-2.0 dimensions. The results are normalized per dimension for a clearer comparison. For detailed numerical results, refer to Table 2.

🔼 This figure presents a statistical overview of the prompts used in the VBench-2.0 benchmark suite. The left panel displays a histogram showing the distribution of word counts across all prompts. This reveals the overall length and complexity of the prompts used to evaluate different aspects of video generation. The right panel shows a bar chart illustrating the number of prompts designed for each of the 18 fine-grained evaluation dimensions within the VBench-2.0 framework. This provides insights into the relative emphasis placed on each dimension during the evaluation process.

read the caption

Figure 3: Overview of Prompt Suite Statistics. Left: distribution of words in the prompt suites. Right: number of prompts per evaluation dimension.

🔼 This figure shows the user interface for human preference annotation in the VBench-2.0 benchmark. The top section displays the question description, the right section lists the choices available for the annotators, and the bottom-left section shows controls for starting, stopping, and playing back the videos.

read the caption

Figure 4: Interface for Human Preference Annotation. Top: Question descriptions. Right: Choices available to annotators. Bottom left: Controls for stopping and playback.

🔼 This figure displays the correlation between human judgment and VBench-2.0 evaluation results across 18 different video generation capabilities. Each plot shows the win ratios from human preference tests (horizontal axis) and VBench-2.0 automated evaluation (vertical axis) for a given video generation model. A linear fit and Spearman’s correlation coefficient (ρ) are calculated for each dimension to quantitatively assess the agreement between human and automated evaluations. The results demonstrate a high correlation, suggesting that VBench-2.0 aligns well with human judgment across all the tested dimensions.

read the caption

Figure 5: Human Alignment of VBench-2.0 Evaluation. Each plot represents the alignment verification for a specific VBench-2.0 dimension. In each plot, a dot corresponds to the human preference win ratio (horizontal axis) and the VBench-2.0 evaluation win ratio (vertical axis) for a given video generation model. A linear fit is applied to visualize the correlation, and Spearman’s correlation coefficient (ρ𝜌\rhoitalic_ρ) is computed for each dimension. Experiments show that VBench-2.0 evaluations closely align with human judgement in all dimensions.

🔼 This figure shows an example of the evaluation for the Mechanics sub-dimension within the Physics dimension of the VBench-2.0 benchmark. It illustrates how the benchmark assesses whether video generation models adhere to basic mechanical principles. The question is posed: ‘Whether the soda can be squeezed as air is gradually and forcefully removed.’ Then it shows two possible video generation results, one where the soda can remains unchanged (score: 0) and one where it is compressed (score: 1), demonstrating the evaluation criteria used to assess the model’s understanding of mechanical properties like compression and force.

read the caption

Figure S6: Example for Mechanics.

🔼 The figure shows an example of the Material evaluation dimension in VBench-2.0. It presents a question and two possible answers to assess a video generation model’s ability to simulate material properties realistically. The question is about whether a grey color results from mixing equal amounts of white and black paint. Option (a) shows the incorrect outcome (no grey), while option (b) displays the correct outcome (yes, grey). This tests the model’s understanding of basic material interactions.

read the caption

Figure S7: Example for Material.

🔼 This figure shows an example of the Thermotics evaluation dimension in VBench-2.0. The prompt asks whether dry ice will remain solid and maintain its original shape at -90°C. Option (a) shows a video where this is not the case, receiving a score of 0. Option (b) illustrates a video where the dry ice behaves correctly, resulting in a score of 1. This highlights the benchmark’s ability to assess a model’s understanding of material properties and their changes under varying temperature conditions.

read the caption

Figure S8: Example for Thermotics.

🔼 This figure shows an example of the Multi-View Consistency evaluation from VBench-2.0. It visually demonstrates how the benchmark assesses whether a video maintains geometric consistency, especially when there is fast camera motion. The figure likely contrasts videos where objects maintain their shape and spatial relationships despite camera movement against those where they do not.

read the caption

Figure S9: Example for Multi-View Consistency.

🔼 This figure demonstrates the diversity aspect of the VBench-2.0 benchmark. It shows two example videos generated by different models in response to the same prompt. The goal is to illustrate how diverse the outputs can be when models try to fulfill a prompt, and that diversity is one way to measure creativity in video generation. Visual differences in style and content composition highlight the varying results.

read the caption

Figure S10: Example for Diversity.

🔼 This figure shows example results for the Composition sub-dimension of the VBench-2.0 benchmark suite. The benchmark evaluates the ability of video generation models to create novel and varied compositions, going beyond simple arrangements. The figure shows examples of video generation model outputs alongside binary evaluations (0 or 1) indicating whether the model succeeded in meeting the specific compositional criteria given in the prompt. These examples illustrate different levels of complexity in composition, such as simple scene arrangements, the combination of multiple entities, and more complex interactions between multiple entities. The goal is to test the model’s ability to create both plausible and novel visual combinations, not just basic arrangements.

read the caption

Figure S11: Example for Composition.

🔼 This figure shows an example of evaluating the Dynamic Spatial Relationship dimension in VBench-2.0. It illustrates a scenario where a dog’s position relative to a sofa changes according to the prompt. The figure displays a sequence of video frames showing the dog initially to the right of the sofa, and then moving to the left of the sofa as instructed by the prompt. This demonstrates the model’s ability to accurately render spatial relationships and object movements in response to detailed textual instructions.

read the caption

Figure S12: Example for Dynamic Spatial Relationship.

🔼 This figure shows an example of evaluating the Dynamic Attribute dimension in VBench-2.0. The Dynamic Attribute dimension assesses whether a model accurately changes the attributes of objects or creatures in a video as specified in a text prompt. The example shown likely illustrates a change in the color of an object. A question is posed: ‘Does the wall change from yellow to grey?’ The image then displays two videos: one where the wall remains yellow (incorrect), and another where the wall changes to grey (correct). This demonstrates the evaluation process; the model’s ability to modify attributes successfully is judged based on whether it reflects the changes described in the prompt.

read the caption

Figure S13: Example for Dynamic Attribute.

🔼 This figure shows an example of evaluating the ‘Motion Order Understanding’ dimension in VBench-2.0. The evaluation involves a video generated by a model in response to a prompt specifying a sequence of actions. The image displays the prompt question and example video frames, illustrating whether the model correctly generated the actions in the specified order. The caption shows two example prompts and two responses for the assessment.

read the caption

Figure S14: Example for Motion Order Understanding.

🔼 This figure shows an example of evaluating the ‘Human Interaction’ dimension in VBench-2.0. It demonstrates the assessment of whether video generation models correctly depict interactions between humans. The specific scenario shown involves one person adjusting another person’s glasses. This necessitates understanding complex human behavior and the spatial relationships between individuals during the action.

read the caption

Figure S15: Example for Human Interaction.

🔼 This figure illustrates the evaluation pipeline and results for the ‘Complex Plot’ dimension of VBench-2.0. It shows how a long, multi-part story prompt is broken down into individual plot points. The video generation model produces a video, and then a Vision-Language Model (VLM) generates a caption describing the video’s content. A Large Language Model (LLM) then compares the VLM’s caption to the original prompt’s plot points to assess how well the video’s narrative matches the intended storyline. The final score reflects the degree of alignment between the generated video and the expected plot progression. The example uses ‘Little Red Riding Hood’ as the story prompt.

read the caption

Figure S16: Example for Complex Plot.

🔼 This figure showcases an example of dynamic camera motion from video generation models. Specifically, it demonstrates how well models can produce specified camera movements, such as zooming in or out. This test is part of the Physics dimension within the VBench-2.0 benchmark suite, which evaluates various aspects of video realism beyond visual aesthetics.

read the caption

Figure S17: Example of Dynamic Camera Motion.

🔼 This figure shows an example of a complex plot evaluation from the VBench-2.0 benchmark. The top shows the prompt used for video generation, detailing a multi-step story of Little Red Riding Hood. Below, the evaluation pipeline is illustrated: the prompt is segmented into five key plot points, a video generative model produces a video, and then LLAVA-video and Qwen models are used to assess how well the video matches each plot point. The final score for plot consistency is displayed.

read the caption

Figure S18: Example for Plot Consistency.

🔼 This figure shows an example of a complex landscape prompt from the VBench-2.0 evaluation suite. It illustrates how well a video generation model can create a long, detailed video that accurately reflects a multi-part description of a landscape. The prompt describes a scene shifting from a high-altitude view of an endless ice plain, progressing through changes in time of day and weather, finally ending with the camera pulling back, revealing the aurora borealis in the vastness of the plains.

read the caption

Figure S19: Example for Complex Landscape.

🔼 This figure shows examples of human anatomy evaluation in VBench-2.0. It visually demonstrates instances where video generation models produce anomalies in human figures, such as missing limbs, distorted body parts, and incorrect hand or face structures. These examples highlight the challenges in achieving high-fidelity human representation in generated videos, a key area assessed by VBench-2.0’s Human Fidelity dimension.

read the caption

Figure S20: Example for Human Anatomy.

🔼 This figure shows an example of evaluating the consistency of human identity in generated videos. Two video clips are presented, one where the identity of the person remains consistent throughout (high score), and another where the identity changes or is inconsistent (low score). This is assessed using the ArcFace model, measuring the similarity of facial features between frames. Inconsistent identity might be indicated by changes in facial appearance, hair style, or other distinguishing features over the duration of the video.

read the caption

Figure S21: Example for Human Identity.

🔼 This figure shows examples of video frames where the human clothes maintain consistency (left) and inconsistency (right) throughout the video. The evaluation assesses if the generated video shows consistent clothing for the human character across all frames, focusing on color, texture, and overall garment. Inconsistent clothing would involve changes in these attributes or instances where the clothing seemingly disappears or changes arbitrarily.

read the caption

Figure S22: Example for Human Clothes.

🔼 This figure shows an example of evaluating the Motion Rationality dimension in VBench-2.0. The image displays a video still where a person is supposedly opening a window. The question posed is whether the person in the video truly completes the action of opening the window, which is a crucial element of assessing intrinsic faithfulness. The ‘yes’ answer implies that the generated video accurately depicts the entire action, demonstrating an understanding of real-world mechanics, while a ’no’ answer indicates a flaw where the motion is incomplete or unrealistic, highlighting a deficiency in the model’s commonsense reasoning or physical simulation capabilities.

read the caption

Figure S23: Example for Motion Rationality.

🔼 This figure demonstrates the evaluation of instance preservation in video generation models. Instance preservation assesses a model’s ability to maintain the correct number of objects throughout a video, even with complex scenarios such as object movement, collisions, and interactions. The figure likely shows example video frames, highlighting instances where a model successfully preserves object counts (correct) and where it fails (incorrect). This allows assessment of the model’s ability to understand and maintain the consistent number of objects within a video scene, even under dynamic changes.

read the caption

Figure S24: Example for Instance Preservation.

🔼 Table 2 presents a comprehensive evaluation of four state-of-the-art video generation models across 18 distinct dimensions defined by the VBench-2.0 benchmark. Each dimension represents a specific aspect of video generation quality, such as human fidelity, controllability, creativity, physics simulation, and commonsense reasoning. The table shows the performance of each model on each dimension, allowing for a detailed comparison of their strengths and weaknesses. Higher scores indicate better performance within that specific dimension.

read the caption

Table 2: VBench-2.0 Evaluation Results per Dimension. This table presents evaluation results for four recent state-of-the-art video generation models across all 18 VBench-2.0 dimensions. A higher score indicates better performance in the corresponding dimension.

🔼 This table provides detailed information about the four state-of-the-art video generation models used in the VBench-2.0 evaluation. For each model, it lists the video length, per-frame resolution, and frame rate (FPS). This information is crucial for understanding the computational resources and characteristics of the videos generated by each model and how these factors might influence the evaluation results.

read the caption

Table 3: Information on Evaluated Models.

🔼 Table S4 presents a detailed comparison of win ratios between automated VBench-2.0 evaluations and human judgments for various video generation model performance across 18 different dimensions. The ‘VBench-2.0 Win Ratio’ column indicates the percentage of times the VBench-2.0 evaluation correctly predicted the superior model based on human preference. Conversely, the ‘Human Win Ratio’ column shows the percentage of times human annotators preferred the model that VBench-2.0 evaluation identified. High similarity between these two ratios for each dimension demonstrates a strong alignment between the automated evaluation and human perception, validating VBench-2.0’s effectiveness.

read the caption

Table S4: Human Alignment of VBench-2.0 Evaluation Methods. For each evaluation dimension and each video generative model, we report “VBench-2.0 Win Ratios (left) / Human Win Ratios (right)”. The results demonstrate that our evaluation metrics closely align with human perception across all dimensions.