ViSpeak: Visual Instruction Feedback in Streaming Videos

🔼 The ViSpeak model architecture is shown. It’s a large multimodal model (LMM) designed for streaming video understanding. It uses separate encoders for audio, video, and text inputs. Notably, it accepts two input streams: one for user inputs (e.g., questions, commands) and another for the model’s own previously generated outputs. These two streams are combined before being fed into a large language model (LLM). A key feature is the inclusion of an ‘informative head’, which enables the model to proactively generate visual outputs, such as initiating conversation based on observed actions in the video stream.

read the caption

Figure 2: ViSpeak is an omni-modality LMM with multiple encoders and a LLM. To support streaming video analysis, ViSpeak takes two input streams as inputs, one for user inputs and one for self-generated outputs. Two streams will be combined into a single one before sending to LLM. An informative head is trained for visual proactive output.

🔼 The figure shows a breakdown of the demographics of the participants involved in recording videos for the ViSpeak-Bench and ViSpeak-Instruct datasets. The data is presented as two charts: a pie chart displaying the gender distribution (approximately equal numbers of males and females), and a bar chart illustrating the age distribution, ranging from 10 to 70 years old.

read the caption

Figure 3: Statistics on participants who recorded videos. The participants comprised nearly equal numbers of males and females, with ages ranging from 10 to 70 years.

🔼 This figure shows a sequence of images from the ViSpeak-Bench dataset that depicts a person performing a visual wake-up gesture. The person is making a sequence of hand gestures, and the figure displays timestamps for each gesture. Below the images, there are annotations showing the corresponding text prompt and the ViSpeak model’s response to that visual input. The response demonstrates the model’s ability to recognize the visual wake-up cue and start a conversation accordingly. This example illustrates the type of data used for the Visual Instruction Feedback task.

read the caption

Figure 4: Examples of Visual Wake-Up in ViSpeak-Bench and the corresponding output by ViSpeak.

🔼 This figure showcases an example from the ViSpeak-Bench dataset illustrating the ‘Visual Termination’ subtask. It displays a video sequence where a user signals the end of a conversation using a nonverbal cue (body language). The annotation details the conversation’s timing and the user’s action. It also shows the corresponding output generated by the ViSpeak model, demonstrating its ability to understand the visual signal and appropriately terminate the interaction. The first round of conversation is included as context for the model’s response.

read the caption

Figure 5: Examples of Visual Termination in ViSpeak-Bench and the corresponding output by ViSpeak. The first round conversation is used as context.

🔼 This figure showcases examples from the ViSpeak-Bench dataset where a visual interruption occurs. The top panel displays a sequence of video frames depicting the interruption, showing the user visually signaling a halt in the conversation. The bottom panel presents the original conversation prompt from the dataset, the model’s initial response before interruption, the user’s visual interruption signal (denoted as ‘(Stop)’), and finally, the model’s concise response to the interruption. This illustrates the model’s ability to recognize and respond appropriately to this non-verbal form of user feedback in real-time.

read the caption

Figure 6: Examples of Visual Interruption in ViSpeak-Bench and the corresponding output by ViSpeak. The first round conversation is used as context.

🔼 This figure showcases examples from the ViSpeak-Bench dataset demonstrating the Gesture Understanding subtask. It visually presents a sequence of frames from a video showing a person making a gesture (a finger heart). Accompanying the video frames is a transcript of the human-agent conversation. The first part of the conversation serves as context for the gesture. The figure then shows the model’s (ViSpeak) response to the gesture, highlighting the model’s ability to understand and interpret the visual cue within the conversational context.

read the caption

Figure 7: Examples of Gesture Understanding in ViSpeak-Bench and the corresponding output by ViSpeak. The first round conversation is used as context.

🔼 This figure showcases two examples from the ViSpeak-Bench dataset demonstrating the ‘Anomaly Warning’ subtask. Each example shows a short video clip depicting an anomalous event (a person falling on a trampoline). The figure displays the corresponding ground truth caption of the video, the response generated by the GPT-4 model, and finally the response generated by the ViSpeak model. The comparison highlights the different abilities of GPT-4 and ViSpeak in handling anomaly detection and generating appropriate warnings.

read the caption

Figure 8: Examples of Anomaly Warning in ViSpeak-Bench and the corresponding output by ViSpeak.

🔼 Figure 9 shows two examples of the Humor Reaction subtask from the ViSpeak-Bench dataset. Each example displays a short video clip with a humorous scene, followed by the ground truth annotation and ViSpeak’s generated response. The annotations provide context and describe the humorous aspects of the video. ViSpeak’s response attempts to capture the humor in a conversational and engaging way, demonstrating its ability to understand and generate appropriate responses to humorous visual content.

read the caption

Figure 9: Examples of Humor Reaction in ViSpeak-Bench and the corresponding output by ViSpeak.

🔼 This figure showcases examples from the ViSpeak-Bench dataset, specifically focusing on the Visual Reference subtask. It presents several short video clips where a user points to an object, and the corresponding response from the ViSpeak model. The goal of the subtask is to test the model’s capability to understand visual references within videos. The figure highlights the accuracy of ViSpeak in correctly identifying the referenced object based on the user’s visual cue, demonstrating its ability to process visual information and respond accordingly within the context of the video.

read the caption

Figure 10: Examples of Visual Reference in ViSpeak-Bench and the corresponding output by ViSpeak.

🔼 This figure showcases examples from the self-annotated dataset used for gesture understanding. It displays video segments with annotations showing both the gesture itself and the intended meaning or context of that gesture within a conversation. The annotations help train the model to accurately interpret and respond appropriately to various gestures. These examples illustrate the diversity of gestures and their accompanying explanations present in the dataset.

read the caption

Figure 11: Examples of our self-annotated data for gesture understanding.

🔼 Figure 12 showcases three instances where the ViSpeak model’s responses were inaccurate. The first example, ‘Time Mistake,’ illustrates a situation where the model generates a response at an inappropriate moment in the video, unrelated to the on-screen activity. The second example, ‘Content Mistake,’ demonstrates a failure to correctly understand the visual content of the video. The model’s response misinterprets what is happening in the video. The third example, ‘Context Mistake,’ highlights the model’s inability to grasp the context of the conversation, leading to an irrelevant and inaccurate response.

read the caption

Figure 12: Examples of failure cases. The ‘Time Mistake’ denotes the model responds at an improper time. The ‘Content Mistake’ denotes the model fails to understand the visual content in the video. The ‘Context Mistake’ means the model is unaware of the context of the conversation.

🔼 ViSpeak-Bench is a benchmark dataset designed for evaluating models’ ability to perform Visual Instruction Feedback. It contains 1000 videos and 1000 question-answer pairs across seven subtasks: Visual Wake-Up, Anomaly Warning, Gesture Understanding, Visual Reference, Visual Interruption, Humor Reaction, and Visual Termination. Each subtask focuses on a specific aspect of visual instruction understanding in streaming video, encompassing various interactive scenarios.

read the caption

Table 2: ViSpeak-Bench benchmark statistics. ViSpeak-Bench contains 7 subtasks with 1,000 videos and 1,000 QA pairs.

🔼 This table details the distribution of samples across different subtasks within the ViSpeak-Instruct dataset. It shows how many samples are dedicated to each of the seven subtasks (Visual Wake-Up, Anomaly Warning, Gesture Understanding, Visual Reference, Visual Interruption, Humor Reaction, and Visual Termination) and indicates whether the data source is self-collected or from other online/offline datasets. The ratio of samples for each subtask relative to the overall dataset is also provided. This information is crucial for understanding the composition and balance of the dataset used to train and evaluate the ViSpeak model.

read the caption

Table 3: Task and sample distribution in ViSpeak-Instruct.

🔼 This table presents the performance of the ViSpeak model on the StreamingBench benchmark [29], a widely used evaluation set for assessing real-time visual understanding capabilities. The results shown are specifically for the ViSpeak model after undergoing both the second and third stages of fine-tuning. This allows for a comparison of the model’s performance at different stages of training and highlights improvements gained through subsequent training phases. The table enables an assessment of ViSpeak’s performance relative to other models on various subtasks within the StreamingBench, including aspects of omni-source understanding and contextual understanding.

read the caption

Table 4: Performance on StreamingBench [29]. Results for ViSpeak trained after the second and third stage are reported.

🔼 This table presents a comparison of the performance of various large multi-modal language models (MLLMs) on the OVO-Bench benchmark [27]. OVO-Bench assesses streaming video understanding capabilities focusing on backward tracing, real-time visual perception, and forward active responding. The table highlights key performance metrics for each model, with a focus on ViSpeak’s results after the second stage of its three-stage fine-tuning process. This allows for evaluation of ViSpeak’s performance compared to other state-of-the-art models in terms of its streaming video understanding capabilities.

read the caption

Table 5: Performance of various MLLMs on OVO-Bench [27]. Results for ViSpeak trained after the second stage are reported.

🔼 Table 6 presents a comprehensive comparison of various large multimodal language models (MLLMs) on the ViSpeak-Bench benchmark. The table showcases the performance of different models across seven distinct subtasks of the Visual Instruction Feedback task, including Visual Wake-up, Anomaly Warning, Gesture Understanding, Visual Reference, Visual Interruption, Humor Reaction, and Visual Termination. The results displayed are for the ViSpeak model after completing its third and final fine-tuning stage, representing a state-of-the-art (SOTA) performance on the benchmark. To establish a baseline for comparison, human evaluation is also included. Five participants, without prior training on the dataset, answered 20% of the questions randomly selected from ViSpeak-Bench. Both the average and maximum scores from these human evaluators are reported for each subtask, providing a direct comparison to the MLLM results and highlighting the challenges inherent in the visual instruction feedback task.

read the caption

Table 6: Performance of various MLLMs on ViSpeak-Bench. Results for ViSpeak trained after the third stage are reported. For human evaluation, we invite 5 participants which are not received relevant training to answer 20% randomly selected questions and we report their average scores and the maximum scores on each subtask.

🔼 This ablation study investigates the impact of different methods for combining user input and model-generated output streams in a streaming video understanding model. It compares three approaches: Adaptive Sum (weighted averaging), Linear (concatenation followed by linear transformation), and Add (direct addition). The results, measured by metrics such as MME (Multi-Modal Evaluation) and scores on MVBench and Video-MME benchmarks, help determine the optimal strategy for integrating dual streams effectively to enhance model performance.

read the caption

Table 7: Ablation studies on input stream combination methods

🔼 This ablation study investigates the design of visual proactive speaking control within a streaming video understanding model. The table presents results from experiments on the StreamingBench benchmark, comparing different approaches to controlling when the model speaks based on visual cues. Specifically, it analyzes the impact of using a language modeling head versus an informative head for prediction, whether the head is jointly fine-tuned with the large language model (LLM), and which token (e.g., visual segment token, end-of-segment token) serves as input for the prediction mechanism. The metrics reported likely include performance scores across various sub-tasks within the StreamingBench.

read the caption

Table 8: Ablation studies on the design of visual proactive speaking control. Performances on StreamingBench are reported. ‘Head’ denotes using language modeling head or informative head for prediction. ‘Joint’ denotes whether the head is finetuned with LLM. ‘Token’ means which token is used for prediction.

🔼 This table presents the results of ablation studies conducted to evaluate the impact of offline data on the ViSpeak-Instruct dataset. Specifically, it shows how the performance of the ViSpeak model changes on the ViSpeak-Bench benchmark when trained with and without offline data. The metrics reported likely include scores for various subtasks within the ViSpeak-Bench (such as gesture understanding, anomaly warning, etc.). Comparing these scores reveals the contribution of offline data to the model’s overall performance, particularly highlighting any improvement or decline in specific areas.

read the caption

Table 9: Ablation studies on the offline data in ViSpeak-Instruct. Performance on ViSpeak-Bench are reported.

🔼 This table presents a comparison of the ViSpeak model’s performance across three training stages on various benchmarks. It shows the model’s performance on StreamingBench, MVBench, Video-MME, and ViSpeak-Bench, highlighting how performance improves with each subsequent training stage. The results for ViSpeak in the second and third training stages are directly compared to the performance of the model in the first training stage as well as other models mentioned in the paper.

read the caption

Table 10: Performance on different benchmarks across different training stages. Results in purple are reported in above tables.

🔼 This table lists the licenses associated with the source datasets used in the creation of the ViSpeak-Bench and ViSpeak-Instruct datasets. It clarifies the usage rights for each dataset, indicating whether they are licensed under Creative Commons (CC BY-NC-SA 4.0), MIT License, or have no specified license (N/A). This information is crucial for understanding the permissible uses and restrictions of the data included in the ViSpeak project.

read the caption

Table 11: License of source datasets in ViSpeak-Bench and ViSpeak-Instruct.