BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

🔼 BlueLM-V-3B’s architecture is a modified version of the LLaVA approach. It consists of an image encoder (SigLIP-400M), an MLP projector, and a language model (BlueLM-2.7B). To handle high-resolution images efficiently, a dynamic resolution processing module is included, similar to those used in LLaVA-NeXT and InternVL 1.5. A token downsampler is added to reduce the number of tokens, which improves efficiency for mobile devices. The diagram shows how images and text are processed and passed to the language model for response generation.

read the caption

Figure 2: Model architecture of BlueLM-V-3B. The architecture of BlueLM-V-3B follows the classical LLaVA approach. We integrate a dynamic resolution processing module (as in LLaVA-NeXT [70] and InternVL 1.5 [22]) to enhance model capabilities and apply token downsampling to reduce deployment complexity.

🔼 This figure compares the image processing approaches of three different Multimodal Large Language Models (MLLMs): LLaVA-NeXT, InternVL 1.5, and the authors’ proposed BlueLM-V-3B. It highlights how each model handles high-resolution images. LLaVA-NeXT and InternVL 1.5 both utilize dynamic resolution schemes but tend to significantly enlarge images, leading to a larger number of image tokens after processing by the Vision Transformer (ViT). LLaVA-NeXT increases the image area by 4 times, while InternVL 1.5 increases it by 25 times. In contrast, BlueLM-V-3B uses a fixed 1:1 aspect ratio, minimizing image enlargement and resulting in the fewest image tokens. This optimized approach leads to more efficient model training and deployment on mobile devices.

read the caption

Figure 3: Existing methods overly enlarge images. (A) For LLaVA-NeXT, an image with resolution 394×\times×390 selects a 2:2 aspect ratio and is resized and padded to 768×\times×768 (4×\times× area enlargement). (B) For InternVL 1.5, an image with resolution 380×\times×76 chooses a 5:1 aspect ratio and is directly resized to 1920×\times×384 (25×\times× area enlargement). BlueLM-V-3B, in contrast, selects a 1:1 aspect ratio for both resolutions, resulting in the minimum number of image tokens after ViT encoding, which can facilitate both model training and deployment.

🔼 This figure illustrates the parallel processing of image patches on the Neural Processing Unit (NPU) of a mobile device, a key optimization in BlueLM-V-3B. The image shows four patches being processed concurrently using the batched image encoding approach, significantly improving processing speed. This contrasts with sequential processing of patches, which would be much slower. The system utilizes a pipeline to take advantage of the NPU’s capabilities and minimize latency.

read the caption

Figure 4: Batched image encoding on NPU. We design a parallel processing scheme for image patches on the NPU. The figure illustrates the case of 4 patches being processed in parallel.

🔼 This figure illustrates how pipeline parallelism and batched image encoding are used in BlueLM-V-3B to speed up image processing. The process begins with multiple image patches from a single image, which are encoded in parallel using the Conv2D layer of SigLIP on the CPU. These intermediate results then feed into the Vision Transformer blocks on the NPU for further parallel processing, significantly shortening the overall inference time.

read the caption

Figure 5: Pipeline parallelism in image encoding. We design a pipeline parallelism scheme for image encoding. The Conv2D layer in the vision embedding module of SigLIP (on the CPU) and the vision transformer blocks (on the NPU) for different image patches run parallel to improve inference speed. This image illustrates the pipeline parallelism scheme combined with batched image patch encoding.

🔼 The figure illustrates the overall framework of the BlueLM-V-3B deployment. It highlights a key efficiency improvement: decoupling the image processing (handled by the ViT) from user input processing (text or audio instructions). This allows parallel processing, where image encoding happens concurrently with the handling of user instructions. Once the image encoding is finished, the user instruction is submitted to the LLM for response generation. For added user-friendliness, the generated text responses can be converted into audio responses in real-time.

read the caption

Figure 6: Overall framework of deploying BlueLM-V-3B. We decouple ViT image processing from user instruction (text or audio) handling to enhance overall efficiency. The text responses by LLM can be further converted on the fly to audio responses.

🔼 This figure shows the inference time of the Vision Transformer (ViT) model when processing image patches with a 2:4 aspect ratio. The experiment varies the number of image patches processed per batch on the Neural Processing Unit (NPU): 1, 2, 4, and 6. Each batch consists of a global patch and 8 local patches. The results show that processing 4 patches per batch achieves the fastest inference time, indicating an optimal balance between parallelization and computational overhead.

read the caption

Figure 7: ViT inference time for 2:4 resolution aspect ratio. We experiment with 1, 2, 4, and 6 image patches per batch on the NPU, using a 2:4 resolution aspect ratio (comprising one global patch and 8 local patches). Overall, processing 4 patches per batch delivers the fastest performance.

🔼 Figure 8 illustrates the trade-off between latency and throughput when processing various numbers of input tokens concurrently in the BlueLM-V-3B model. The x-axis represents the number of tokens processed in parallel (t{x}/t1 denotes processing x tokens in parallel), while the y-axis shows both latency (in seconds) and output speed (in tokens per second). The figure highlights that increasing the number of parallel tokens initially reduces latency and increases throughput, but beyond a certain point, this trend reverses likely due to the limitations of NPU resources. The output token count remains fixed at one per forward pass, independent of the number of parallel tokens processed, reflecting the autoregressive nature of the LLM. This emphasizes the efficiency optimization achieved in BlueLM-V-3B.

read the caption

Figure 8: Latency and output speed comparison. We compare the latency and output generation speed with processing different numbers of input tokens in parallel. t{x𝑥xitalic_x}/t1 implies processing x𝑥xitalic_x input tokens in parallel. The output token is fixed to 1 per trunk as the LLM can only generate one token for each forward process.

🔼 This figure demonstrates the exaggerated image resolution in existing methods. Panel (A) shows that LLaVA-NeXT chooses a resolution of 384x768 for an image originally sized 380x393, significantly increasing the image size. Panel (B) illustrates InternVL 1.5 selecting a resolution of 1920x384 for an image initially sized 500x102, further highlighting the issue of excessive enlargement. This excessive enlargement increases the number of image tokens, hindering efficient deployment on mobile devices.

read the caption

Figure 9: Case study. (A) LLaVA-NeXT chooses resolution 384×\times×768 for an image with the original size of 380×\times×393. (B) InternVL 1.5 chooses resolution 1920×\times×384 for an image with the original size of 500×\times×102.

🔼 This table compares the performance of different dynamic image resolution methods used in training multimodal large language models (MLLMs). The comparison uses two models with similar parameter counts: the in-house BlueLM-3B and the open-source MiniCPM-2B (both around 2.7B parameters). The LLaVA dataset (558k for pre-training and 665k for fine-tuning) was used for training. The table highlights the superior performance of the proposed dynamic image processing method in BlueLM-V-3B, and also includes results for a fully trained BlueLM-V-3B model for additional context.

read the caption

Table 2: Comparison results of different dynamic resolution methods. We compare the performance of models trained using different dynamic resolution methods. We use the LLaVA [69] 558k dataset for pre-training, and the LLaVA 665k dataset for fine-tuning. To better demonstrate our improvements, we conduct experiments on both our in-house BlueLM-3B language model and the open-sourced MiniCPM-2B language model, which have similar parameter counts (2.7B). Our dynamic image processing method achieves the best performance. †We also provide the results of the fully trained BlueLM-V-3B model for reference.

🔼 This table presents a comparison of BlueLM-V-3B’s performance against other large language models (LLMs) on the OpenCompass benchmark. OpenCompass is a comprehensive evaluation suite for foundation models, assessing performance across a range of tasks. The table focuses on models with 10 billion parameters or fewer. BlueLM-V-3B, despite having only 3 billion parameters, achieves state-of-the-art results on four out of the eight tasks evaluated and ranks second overall.

read the caption

Table 3: OpenCompass benchmark. Comparison results on the OpenCompass benchmark for models with parameter sizes less than or equal to 10B. BlueLM-V-3B achieves state-of-the-art performance on 4 out of 8 tasks, with an average performance ranking of second.

🔼 Table 4 presents a comparison of BlueLM-V-3B’s performance on text-centric and OCR benchmarks against other state-of-the-art (SOTA) Multimodal Large Language Models (MLLMs). The focus is on models with similar parameter sizes. The results show BlueLM-V-3B achieves comparable performance to these SOTA models but with a significant advantage in multilingual capabilities. To ensure fairness, TextVQA and MTVQA evaluations used the VLMEvalKit [29]. Note that OCRBench results are included within the OpenCompass benchmark.

read the caption

Table 4: Text-centric/OCR benchmarks. Comparison on text-centric/OCR benchmarks shows that BlueLM-V-3B achieves performance comparable to SOTA MLLMs with similar parameter sizes, while significantly enhancing multilingual capability. †We evaluate TextVQA and MTVQA on VLMEvalKit [29] for a fair comparison. OCRBench has been included in OpenCompass.

🔼 This table compares the deployment efficiency of BlueLM-V-3B with MiniCPM-V. MiniCPM-V uses an 8B parameter model and runs on the CPU, resulting in significantly slower image processing, LLM pre-filling (preparing the language model for generation), and overall throughput (tokens generated per second) compared to BlueLM-V-3B. The difference is attributed to MiniCPM-V’s CPU deployment and the inclusion of model loading time in its latency calculation. BlueLM-V-3B’s superior efficiency stems from its smaller 3B parameter model and use of the NPU (Neural Processing Unit). The 0.47s load time for BlueLM-V-3B accounts for the simultaneous loading of both Vision Transformer (ViT) and Language Model (LLM) components at the start of the system initialization.

read the caption

Table 5: Deployment efficiency comparison with MiniCPM-V. MiniCPM-V deploys an 8B model on the CPU, leading to longer image processing latency, LLM prefilling latency, and lower throughput. †MiniCPM-V calculates encoding latency by including both model loading time and encoding time. In our setting, we need 0.47s to simultaneously load the ViT and LLM once during system initialization.

🔼 Table 6 lists the open-source datasets used in the fine-tuning stage of the BlueLM-V-3B model training. It details the datasets used for each task category (Text-only, Caption, OCR, and VQA), showing the specific datasets contributing to each category. The table connects these datasets to the data volumes reported in Table 1 of the paper, providing context for the scale of the fine-tuning data used. Note that some datasets may be used in multiple categories.

read the caption

Table 6: Training data. This table presents the open-source datasets used in the fine-tuning stage, corresponding with the categories and data volume in Tab. 1 of the main text.

🔼 This table details the hyperparameters used during the pre-training phase (stage 1) of the BlueLM-V-3B model. It includes settings for the optimizer (AdamW), learning rate schedule (cosine decay), weight decay, training steps, warm-up steps, batch size, gradient accumulation, and numerical precision. These settings are crucial for controlling the training process and achieving optimal model performance.

read the caption

Table 7: Hyper-parameters. Hyper-parameters for the pre-training stage (stage 1).

🔼 This table details the hyperparameters used during the fine-tuning stage of the BlueLM-V-3B model training. It includes settings for the optimizer (AdamW), learning rate schedule (cosine decay), weight decay, batch size, and other relevant parameters. Note that because of upsampling for some smaller datasets, the total number of training steps and the batch size might exceed the total volume of data.

read the caption

Table 8: Hyper-parameters. Hyper-parameters for the fine-tuning stage (stage 2).