Baichuan-Omni-1.5 Technical Report

🔼 Baichuan-Omni-1.5 is a multimodal model that can process various input modalities (text, audio, image, and video) and generate corresponding outputs (text and audio). The figure details the model’s architecture. It shows how visual and audio information is encoded and fed into a shared LLM decoder. This decoder alternately generates text and audio tokens. The audio tokens are then passed to an audio decoder to generate the final audio output. This allows for end-to-end audio generation and seamless integration across modalities.

read the caption

Figure 2: Architecture of Baichuan-Omni-1.5 . Our model is designed to process both pure text/audio inputs and combinations of video/image with text/audio. When generating audio, the Baichuan-Omni-1.5 LLM Decoder alternately predicts text tokens and audio tokens. The audio tokens are then decoded by the Audio Decoder to produce the final audio.

🔼 Figure 3 illustrates the diverse multimodal data used to pre-train Baichuan-Omni-1.5. The dataset includes various data types: pure text, image-text pairs, video-text pairs, and audio-text pairs. Importantly, it also incorporates more complex interleaved data formats, such as image-audio-text triplets and video-audio-text triplets. This rich and varied dataset allows the model to learn comprehensive cross-modal relationships and achieve a high level of multimodal understanding.

read the caption

Figure 3: Pretrain Data illustration of Baichuan-Omni-1.5 . We construct an extensive omni-modal dataset, including text, image-text, video-text, audio-text, and their interactions. Our collection also contains interleaved image-audio-text and video-audio-text data.

🔼 This figure illustrates the architecture of the Baichuan-Omni-1.5 model’s audio branch. It uses a Residual Vector Quantization (RVQ) based audio tokenizer to convert raw audio into discrete tokens, which are then processed by a Large Language Model (LLM). The audio tokens are further decoded into a speech waveform using a flow-matching model based on a U-Net architecture and a HiFi-GAN vocoder. The model utilizes a multi-stage training strategy to improve speech quality, incorporating techniques such as multi-scale Mel loss. The figure highlights the different components of the audio processing pipeline and their interactions, showing how raw audio is transformed into meaningful representations understandable by the LLM, then ultimately converted back into speech.

read the caption

Figure 4: Audio tokenizer and audio decoder based on flow matching model.

🔼 This figure illustrates the training pipeline of the Baichuan-Omni-1.5 model. The training process is divided into four stages. Stage 1 pre-trains the model on image-text data to enable visual understanding. Stage 2 incorporates audio using a newly designed audio tokenizer and embedding layer, enabling end-to-end audio processing. Stage 3 integrates image, audio, video, and text data to achieve comprehensive omni-modal understanding, extending the model’s maximum sequence length for handling longer inputs. Finally, Stage 4 fine-tunes the model through supervised learning using omni-modal data, with specific sub-stages focusing on improving both instruction following and audio generation capabilities.

read the caption

Figure 5: Training Pipeline of Baichuan-Omni-1.5 . The pretraining phase is divided into three stages to incrementally incorporate vision and audio into the LLM while relieving modality conflicts. Stage 1 focuses on image-text training, which extends an LLM to process and understand visual input. Stage 2 extends an LLM pre-trained on visual data to understand audio input in end-to-end manner by incorporating our Baichuan-Audio-Tokenizer, a newly introduced audio embedding layers and an independent audio head. Stage 3 focuses on training Baichuan-Omni-1.5 using high-quality cross-modal interaction datasets encompassing image-audio-text and video-audio-text format, and extends the maximum sequence length to 64k to support long audio and video stream. Stage 4 enhances the model’s instruction following and audio capabilities through supervised fine-tuning with omni-modal data. Stage 4.1: Freeze the Audio Head using omni-modal understanding data to boost modality interactivity and multitasking comprehension. Stage 4.2: Activate only the Audio Head and Audio Embed layer, with audio generation data to improve speech generation capabilities.

🔼 This table presents a detailed breakdown of the video training data used in the Baichuan-Omni-1.5 model. It categorizes the data into question-answering (QA) and description types, specifying the dataset source (public or synthetic), and the number of questions and descriptions available for training.

read the caption

Table 2: Detailed statistics of the training data of video pretrain.

🔼 This table presents a detailed breakdown of the audio training data used in the Baichuan-Omni-1.5 model. It shows the different types of audio data used (audio understanding and audio generation), their specific tasks (ASR, AQA, etc.), the data format, and the total number of hours of audio data for each task. This information highlights the scale and diversity of the audio data used to train the model’s audio capabilities.

read the caption

Table 3: Detailed statistics of the training data of audio pretrain.

🔼 Table 4 presents a summary of the dataset used for the supervised fine-tuning stage of the Baichuan-Omni-1.5 model. It details the quantity of data in various modalities (text, image, video, audio) and combinations thereof that were used to train the model to perform more effectively on a wider range of complex tasks involving multiple modalities. The table breaks down the data by category and provides counts to illustrate the scale and diversity of the training data.

read the caption

Table 4: Omni-modal SFT data statistics for Baichuan-Omni-1.5 . Here we summarize the category and quantities of our SFT dataset.

🔼 Table 5 details the composition of the image data used for supervised fine-tuning of the Baichuan-Omni-1.5 model. It breaks down the data by task category (GeneralQA, OCR, Graphical, Mathematics, Spatiotemporal, Captioning, Medical), listing the specific datasets contributing to each category and the percentage of the overall image SFT dataset each category represents. This provides insight into the diversity and distribution of training data used to enhance the model’s capabilities in various visual understanding tasks.

read the caption

Table 5: Image SFT data for Baichuan-Omni-1.5 . This table summarizes the image SFT dataset categories, their sources, and proportions for various tasks.

🔼 Table 6 presents a comprehensive comparison of various large language models (LLMs) on several prominent pure text benchmarks. It includes both proprietary models (those developed by companies and not publicly available) and open-source models (those available for public use). The benchmarks used assess general knowledge, reasoning abilities, and Chinese language understanding. Results are shown as accuracy scores, with distinctions made for officially reported scores, scores retrieved from leaderboards, and scores reproduced by the authors. This allows for a nuanced comparison of the models’ capabilities across different assessment types.

read the caption

Table 6: Results on comprehensive pure text benchmarks. ∗*∗: Officially reported results. ♢♢\diamondsuit♢: Retrieved results from official leaderboard or recent papers. Other unlabeled results are reproduced by ourselves.

🔼 Table 7 presents a comparison of various models’ performance on multiple-choice and yes/no question benchmarks related to image understanding. The models are categorized into proprietary (closed-source) models and open-source models (further divided into vision-language models and omni-modal models). The benchmarks used assess accuracy across different visual question answering tasks, reflecting the models’ abilities in understanding image content and responding accordingly. Results marked with ∗ are officially reported scores, while those marked with ♢ are obtained from official leaderboards or recent papers. The remaining results were reproduced by the authors of the paper for fair comparison.

read the caption

Table 7: Results on Multi-choice benchmarks and Yes-or-No benchmarks. ∗*∗: Officially reported results. ♢♢\diamondsuit♢: Retrieved results from official leaderboard or recent papers. Other unlabeled results are reproduced by ourselves.

🔼 Table 8 presents a comparison of various models’ performance on several Image Visual Question Answering (VQA) benchmarks. The benchmarks assess a model’s ability to understand and answer questions about images. The table includes both proprietary models (those developed and maintained by companies like OpenAI) and open-source models (publicly available models). For each model, the table shows the accuracy achieved on each benchmark. Results marked with a * are officially reported by the model’s developers, while those marked with a ♢ were taken from official leaderboards or recent research papers. The remaining results were reproduced by the authors of this paper to ensure consistent comparison. This allows for a fair comparison across different models, considering potential variations in testing conditions.

read the caption

Table 8: Results on image VQA benchmarks. ∗*∗: Officially reported results. ♢♢\diamondsuit♢: Retrieved results from official leaderboard or recent papers. Other unlabeled results are reproduced by ourselves.

🔼 Table 9 presents the results of general video question answering (VQA) benchmarks. It compares various models’ performance across four metrics: MVBench, Egoschema, VideoMME, and Perception-Test. The table differentiates between proprietary models (those developed by companies and not publicly released), open-source vision-language models, and open-source omni-modal models. Results are shown as accuracy (%). The maximum number of frames sampled for evaluation is also specified. Note that a specific setting (’no subtitles’) was used for the VideoMME benchmark. Results marked with * represent officially reported scores from the original papers or leaderboards. Results with ♢ represent scores obtained from official leaderboards or recent papers. The remaining scores were reproduced by the authors of this paper using consistent evaluation settings for fair comparison.

read the caption

Table 9: Results on general video VQA benchmarks. max: Maximum number of sampling frames. ∗*∗: Officially reported results. ♢♢\diamondsuit♢: Retrieved results from official leaderboard or recent papers. Other unlabeled results are reproduced by ourselves. Note that we use the 'no subtitles' evaluation setting in VideoMME.

🔼 Table 10 presents the results of open-ended Video Question Answering (VQA) benchmark evaluations. It compares the performance of several models, including proprietary models (Gemini 1.5 Pro, GPT-4 mini, GPT-4, GPT-4V) and open-source models (Qwen2-VL, VideoLLaMA 2, VideoChat2, LLaVA-NeXT-Video, Video-LLaMA, VITA, VITA-1.5, Baichuan-Omni, MiniCPM-0 2.6, Baichuan-Omni-1.5). The models are evaluated on two benchmarks: ActivityNet-QA and MSVD-QA. The table shows the accuracy (Acc.) and score achieved by each model on both benchmarks. The number of frames sampled from each video is also indicated, with some models using a maximum number of frames and others a fixed frames per second.

read the caption

Table 10: Results on open-ended video VQA benchmarks. max: Maximum number of sampling frames. ∗*∗: Officially reported results. Other unlabeled results are reproduced by ourselves.

🔼 Table 11 presents the results of evaluating various models on several audio understanding benchmarks. The benchmarks assess different aspects of audio comprehension, including reasoning, question answering, and general knowledge. The table compares the performance of proprietary models (GPT-40-Audio) and open-source models (GLM-4-Voice, VITA-1.5, MiniCPM-0 2.6, Baichuan-Omni-1.5) across these benchmarks. Different evaluation methods are employed, as denoted by symbols in the caption: ∇∇\nabla∇ indicates text-only evaluation on text-audio interleaved outputs; ♢♢\diamondsuit♢ specifies text-audio interleaved output only; and □□\square□ denotes a cascaded output approach, with evaluation based on the produced text. This allows for a multifaceted comparison of the models’ abilities in various audio understanding tasks.

read the caption

Table 11: Results on audio understanding benchmarks. ∇∇\nabla∇: The modalities parameter is set to ['text', 'audio'], evaluation based on the output text. ♢♢\diamondsuit♢: Supports only text-audio interleaved output. □□\square□: Cascade output method, evaluation based on the output text.

🔼 Table 12 presents a comprehensive comparison of the overall omni-understanding capabilities of several large language models (LLMs), including both proprietary models (GPT-40-mini) and open-source omni-modal models. The models are evaluated across four distinct scenarios, each involving a unique combination of input modalities: Image & Audio, Image Caption & Audio, Image & Audio Transcript, and Image Caption & Audio Transcript. The table highlights the accuracy scores achieved by each model in each scenario, offering a nuanced assessment of their cross-modal understanding performance. Note that for the GPT-40-mini model, audio input was not directly supported; therefore, audio data was transcribed before input for fair comparison.

read the caption

Table 12: Overall Omni-Undesratnding Results. All the results are reproduced by ourselves. GPT-4o-mini does not support audio input, we use its audio API and transcribe the audio and then input it.

🔼 Table 13 presents the performance comparison of various models on medical image understanding benchmarks. The benchmarks used are GMAI-MMB-VAL and OpenMM-Medical. The results shown represent accuracy scores achieved by each model on each benchmark. All results were reproduced by the authors of the paper for fair comparison. This table is crucial for assessing the models’ capabilities to handle medical image data, a particularly challenging domain requiring high accuracy and reliability.

read the caption

Table 13: Results on medical benchmarks. All the results are reproduced by ourselves.