Video Action Differencing

🔼 The VidDiff method is a three-stage process for identifying subtle differences between two videos showing the same action. First, a large language model (LLM) proposes potential differences between the videos (e.g., ‘faster ascent’, ‘wider stance’). Next, a frame localization module identifies the specific frames in each video where these differences are visible. Finally, a vision-language model compares the identified frames to determine whether each proposed difference applies more strongly to video A, video B, or neither.

read the caption

Figure 2: VidDiff Method. One input is an action description (e.g. “weighted squat”). The Difference Proposer generates potential differences using a large language model (LLM). The Frame Localizer assigns frames where these differences are observable. Finally, the Action Differencer checks each difference using a vision-language model, determining whether it applies more to video A or video B, or neither.

🔼 This figure showcases examples of successful and unsuccessful video action difference predictions made by the GPT-40 model. The ‘success cases’ (left) illustrate situations where GPT-40 achieves high accuracy. These cases generally involve readily apparent differences between the videos, easy identification of the specific frames where the differences occur, and actions that are relatively straightforward or simple in nature. In contrast, the ‘failure cases’ (right) represent instances where GPT-40’s performance is poor. These typically involve subtle differences requiring keen visual discernment, precise temporal localization to pinpoint the exact frames containing the difference, or actions that are complex and nuanced. The differences between success and failure highlight the challenges of identifying and localizing nuanced action differences.

read the caption

Figure 3: Examples of ‘success cases’ (left) – differences where GPT-4o has high accuracy – and failure cases (right). Success cases typically involve coarse differences, easy localization, or simple actions, while failure cases often involve fine differences, precise localization or complex actions.

🔼 This figure showcases examples of frame localization within the VidDiff method. It visually compares the ground truth frame selections with the model’s predictions for various action differences. Each row represents a different difference, highlighting the model’s ability to pinpoint the relevant frames where visual differences are observable. The differences shown range in complexity and granularity, demonstrating the challenges in precise temporal localization within video action differencing.

read the caption

Figure 4: Sample frame localizations: prediction vs ground truth.

🔼 This table presents the performance of various large multimodal models (LMMs) on a closed-set video action differencing task. The task involves predicting whether a given difference description applies more to one of two videos of the same action. The table shows the accuracy of each model across three difficulty levels (Easy, Medium, Hard). Scores significantly better than random chance (50%) are highlighted in gray, indicating statistical significance at a p-value below 0.05 using a binomial test. The best performance for each difficulty level is shown in bold, and the second-best is underlined.

read the caption

Table 2: Results for closed setting (accuracy). Best scores in bold, second best underlined. Scores are better than random, with statistical significance highlighted in gray. Significance is p-value<0.05absent0.05<0.05< 0.05 on a binomial test.

🔼 Table 3 presents the results of the open-set evaluation for the Video Action Differencing (VidDiff) task. Recall@Ndiff is used as the evaluation metric, measuring the proportion of correctly identified and generated differences out of the total number of ground truth differences. The table shows the performance of different large multimodal models (LMMs) and the VidDiff method across three difficulty levels: easy, medium, and hard. The best scores for each difficulty level are highlighted in bold, and the second-best scores are underlined.

read the caption

Table 3: Results for open setting (recall@Ndiffsubscript𝑁diffN_{\text{diff}}italic_N start_POSTSUBSCRIPT diff end_POSTSUBSCRIPT). Best scores in bold, second best underlined.

🔼 This ablation study analyzes the performance of the visual question answering (VQA) component of the VidDiff method when provided with perfectly aligned ground truth frames. The experiment tests the ability of zero-shot vision-language models to identify subtle differences between the images. The evaluation is a 3-way multiple-choice question for each comparison.

read the caption

Table 4: Ablation study results for frame differencing VQA with ground truth frames. Questions are 3-way multiple-choice.

🔼 This table presents the ablation study results on the impact of different frame localization techniques on the accuracy of video action differencing. Specifically, it compares the performance of the proposed VidDiff method’s frame localization component against a baseline of random frame retrieval and a method without the Viterbi decoding step. The results are presented for the ’easy’ difficulty split of the VidDiffBench dataset. The table helps illustrate the importance of precise frame localization for accurate video action differencing, showcasing the contribution of the specific techniques employed in VidDiff.

read the caption

Table 5: Ablation on frame localization using different retrieval techniques on easy.

🔼 This table details the difficulty level assigned to each action in the VidDiffBench benchmark dataset. For each action, it provides the action’s short name (code), its full name, and a description of the action itself. The difficulty level is categorized into ’easy’, ‘medium’, or ‘hard’, reflecting the complexity of identifying and annotating subtle differences within the action’s video pairs. This categorization is crucial for evaluating model performance and ensuring the benchmark’s comprehensiveness.

read the caption

Table 6: The difficulty splits with action code and names

🔼 This table presents the results of a human evaluation study conducted to validate the LLM’s approach to assigning difficulty levels (easy, medium, hard) to the video action differencing tasks. Three human evaluators independently ranked the difficulty of the actions, and these rankings were compared against the rankings generated by the LLM using Spearman’s rank correlation. The table shows the correlation coefficients between each human evaluator’s ranking and the LLM’s ranking, offering a quantitative assessment of the agreement between human judgment and the LLM’s automated approach.

read the caption

Table 7: Results on human evaluation study on choosing the splits. This is the Spearman’s rank corrlation between the ranks of action dificulty, comparing our LLM approach and 3 humans.

🔼 This table presents a detailed breakdown of the VidDiffBench dataset statistics, categorized by difficulty level (easy, medium, hard). It includes the number of video pairs, average and total video lengths, the count of difference annotations, and the standard deviation of those annotations within and across retrieval types (indicating the consistency of annotation within a video and across different videos). Finally, it shows the distribution of the A/B/C annotation labels in the dataset, illustrating the relative frequency of different levels of perceived action differences. This granular level of detail helps researchers understand the characteristics of the benchmark and better design and evaluate their models.

read the caption

Table 8: Detailed data statistics by split

🔼 This table presents the agreement rates between the predictions made by a Large Language Model (LLM) and three human annotators on a video action difference matching task. It shows how well the LLM’s automated matching of video action differences aligns with human judgment. The data is presented as a matrix, showing the agreement rates between each pair of the LLM and the three human annotators.

read the caption

Table 9: Agreement rate of LLM and human predictions for the evaluation matching.

🔼 This table shows the pairwise correlation coefficients between the action-level accuracy scores achieved by different large multimodal models (LMMs) on the VidDiffBench benchmark. Higher correlation indicates that the models perform similarly across the different actions in the benchmark. This helps to understand the relationship between the models’ performance and whether they exhibit similar strengths and weaknesses.

read the caption

Table 10: Correlations between models where the data is the action-level accuracy.

🔼 This table presents the performance of five large multimodal models (GPT-40, Gemini-1.5-Pro, Claude-3.5-Sonnet, LLaVA-Video-7B, and Qwen2-VL-7B) on a video action differencing task. It shows the accuracy scores achieved by each model on each individual action within the dataset. The scores are presented for different difficulty levels, and differences between the model’s performance and the average performance across all models for each action are also provided. This allows for a granular analysis of model strengths and weaknesses in this new video understanding task.

read the caption

Table 11: Action-level scores for each model, and their differences compared to the average model score for that action. The model names are abbreviated and the full model names are GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, LLaVA-Video-7B, Qwen2-VL-7B

🔼 This table presents the performance of different large multimodal models (LLMs) on a video action differencing task, broken down by individual actions. Instead of showing absolute scores, it shows the difference between each model’s performance on a specific action and the average performance of all models on that same action. This allows for a more nuanced comparison, highlighting which models excel or struggle on particular actions, rather than just giving overall accuracy metrics. Model names are abbreviated for brevity; the full names are provided in the caption.

read the caption

Table 12: Action-level difference scores for each model relative to the mean model score on that action. This is the difference with respect to the table 11. The model names are abbreviated and the full model names are GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, LLaVA-Video-7B, Qwen2-VL-7B

🔼 This table lists all actions included in the VidDiffBench benchmark dataset, providing a key (identifier) for each action and a detailed description of the action. The descriptions offer sufficient detail to understand the action’s nature and the type of movement involved. This is crucial for researchers working with the dataset, providing them with the necessary context to understand the nuances of the actions being compared.

read the caption

Table 13: Actions keys and their descriptions

🔼 This table presents a detailed breakdown of the accuracy scores achieved for each specific difference identified within the VidDiff benchmark. It shows the accuracy for each difference, the number of samples used in the evaluation for that difference, and whether the accuracy was statistically significant (p-value < 0.05 using a two-tailed binomial test). The differences are linked to their corresponding actions via a key, allowing for cross-referencing with Table 13 which defines the actions. Gray shading highlights statistically significant results.

read the caption

Table 14: Difference-level accuracy scores for VidDiff. The ‘action’ values can be looked up at table 13. The grayed columns indicate a p-value <<< 0.05 for the two-tailed binomial significance test

🔼 This table presents the results of an experiment evaluating the performance of three different large multimodal models (GPT-40, Gemini-1.5-Pro, and Claude-3.5-Sonnet) on the ’easy’ split of the VidDiffBench dataset. The experiment varied the frames per second (fps) of the video input to the models, testing 1 fps, 2 fps, 4 fps, and 8 fps. The evaluation protocol used in the paper establishes 4 fps as the standard. The table shows the accuracy of each model at each fps rate, allowing for comparison of model performance under different input conditions and highlighting the impact of fps on model accuracy.

read the caption

Table 15: Evaluating ‘easy’ split with variable video fps for three models. Our evaluation protocol chooses 4fps.