MLLM-as-a-Judge for Image Safety without Human Labeling

🔼 This figure demonstrates a challenge in using large language models (LLMs) for image safety assessment. A complex safety rule, focusing on images depicting imminent death, is presented to the LLaVA-OneVision-Qwen2-72b-ov-chat model. Despite the image clearly not showing a scene of imminent death, the LLM struggles to correctly assess the image’s safety, highlighting difficulties in applying nuanced and detailed rules.

read the caption

(b) Challenge 2: Current MLLMs struggle to reason with complex, lengthy safety rules. The rule applies to imminent death scenarios, this image clearly does not depict one. The model used here is LLaVA-OneVision-Qwen2-72b-ov-chat (Li et al., 2024).

🔼 This figure demonstrates the inherent biases in large language models (LLMs) when used for image safety assessment. Even when an image doesn’t explicitly show a throat slit (violating a specific safety rule), the LLM (InternVL2-8B-AWQ) might incorrectly flag it as unsafe. This is because the model identifies bloodstains on the ground, leg, and feet, associating them with a throat slit due to existing biases in its training data. This highlights a challenge in directly using LLMs for safety assessments without addressing their inherent biases.

read the caption

(c) Challenge 3: MLLMs have inherent biases. Despite the absence of a throat slit, the MLLM predicts a rule violation due to its bias, linking blood on the ground, foreleg, and feet to a throat slit. Model here is InternVL2-8B-AWQ (Chen et al., 2023).

🔼 This figure illustrates three key challenges in using pre-trained, large multimodal language models (MLLMs) for zero-shot image safety assessment. The task is to determine if an image violates predefined safety rules without fine-tuning the MLLM. The challenges shown are: 1) Subjectivity of safety rules; humans disagree on the appropriateness of an image. 2) Difficulty of MLLMs reasoning with complex, multi-clause rules; the model struggles to correctly apply a detailed safety policy. 3) Inherent biases in MLLMs; the model incorrectly labels a safe image due to bias.

read the caption

Figure 1: Examples showing the challenges for simply querying pre-trained MLLMs for zero-shot image safety judgment.

🔼 This figure illustrates how complex safety rules are broken down into simpler, more manageable ‘precondition chains.’ The example shows a rule prohibiting images depicting people or animals with bloody injuries indicating imminent death. This rule is decomposed into three preconditions: 1) people or animals are visible; 2) visible bloody injuries exist; and 3) injuries appear to cause imminent death. Each precondition is evaluated separately, creating a more straightforward process for the MLLM to assess image safety.

read the caption

Figure 2: Example of the preconditions extracted from the rule.

🔼 This figure illustrates the process of calculating a token-based score to determine whether a precondition is met. The process involves querying a pre-trained Large Language Model (LLM) with a Yes/No question about the precondition’s fulfillment, given an image. The LLM provides probabilities for both ‘Yes’ and ‘No’ responses. A precondition is considered satisfied if the probability of ‘Yes’ divided by the sum of the probabilities of ‘Yes’ and ‘No’ exceeds a predefined threshold.

read the caption

Figure 3: Process of calculating token based score. The precondition is considered satisfied if the score is larger than a threshold.

🔼 This figure illustrates a method to reduce bias stemming from non-centric image regions. The approach involves comparing the token probability-based scores (as described in Figure 3) for two versions of the image: the original and one with the central region removed. A significant difference in scores between the two versions indicates that the bias from the non-centric region is substantial and that the image satisfies the precondition.

read the caption

Figure 4: Approach for mitigating the bias from the non-centric content in the image. We compare the token probability based score (see Figure 3) of the original image and the image with centric region removed, and consider the image satisfy the precondition if the difference of the score is large enough.

🔼 This figure illustrates the cascaded reasoning process used when the initial token probability-based judgment lacks confidence. It shows how the model first attempts a chain-of-thought reasoning process to determine whether a precondition is met. If this process isn’t conclusive enough, it then proceeds to ask the model for a concise summary of its reasoning and prediction in JSON format for better accuracy.

read the caption

Figure 5: Process of cascaded reasoning-based judgment.

🔼 This figure shows the distribution of scores generated by the InternVL2-76B language model for a specific precondition: ‘The visible, bloody injuries indicate imminent death based on the severity of the injury.’ The x-axis represents the score, ranging from 0 to 1, indicating the model’s confidence that the precondition is met. The y-axis shows the percentage of instances receiving each score. The figure likely illustrates the model’s tendency to assign certain scores more frequently than others, possibly due to biases in the training data or the inherent ambiguity of the precondition itself. It helps demonstrate the challenges of directly using large language models for image safety assessment without additional processing or mitigation strategies.

read the caption

(a) Score distribution of InternVL2-76B on the precondition “The visible, bloody injuries indicate imminent death based on the severity of the injury”.

🔼 This figure shows the distribution of scores generated by the InternVL2-26B-AWQ language model when evaluating whether a precondition is met. The precondition in question is: ‘Legs of people in swimwear or underwear are spread apart by an angle exceeding 90 degrees’. The x-axis represents the score, ranging from 0 to 1, where a higher score indicates a higher probability that the precondition is met. The y-axis represents the percentage of preconditions that received a particular score. The chart visualizes the distribution of scores, allowing for an analysis of the model’s confidence in determining whether the given precondition is satisfied for different images. This helps assess the model’s performance and identify potential areas for improvement.

read the caption

(b) Score distribution of InternVL2-26B-AWQ on the precondition “Legs of people in swimwear or underwear are spread apart by an angle exceeding 90 degrees”.

🔼 This figure shows the distribution of scores generated by the Qwen2-VL-72B-Instruct model when evaluating the precondition ‘The visible, bloody injuries indicate imminent death based on the severity of the injury.’ The x-axis represents the score, ranging from 0 to 1, where higher scores indicate a greater likelihood that the precondition is met. The y-axis shows the percentage of images that received a particular score. The distribution is broken down into two categories: ‘Satisfied’ (precondition is met) and ‘Not Satisfied’ (precondition is not met). The figure helps illustrate how well the model can objectively assess the precondition based solely on the visual information in the images.

read the caption

(c) Score distribution of Qwen2-VL-72B-Instruct on the precondition “The visible, bloody injuries indicate imminent death based on the severity of the injury”.

🔼 Figure 6 presents a comparative analysis of score distributions across three different large language models (LLMs) under varying preconditions. For each LLM, two sets of scores are shown: one for images that actually satisfy the precondition, and another for images that do not. Crucially, the figure also displays scores calculated without incorporating image tokens into the model’s input. This ’no image’ condition helps highlight the impact of image features on the model’s output and the magnitude of any biases or language priors influencing the score. Each distribution is displayed as a histogram, showing the frequency of different score values. The x-axis represents the LLM’s score (from 0 to 1), and the y-axis represents the frequency or percentage of instances with a particular score.

read the caption

Figure 6: Score distributions across different models under different preconditions. We show the score distributions for queries containing images with ground-truth label “Satisfied the precondition” and “Not Satisfied the precondition”. Additionally, we illustrate the precondition scores without incorporating image tokens, i.e., ℳ⁢(None,𝒄)ℳNone𝒄\mathcal{M}(\text{None},\bm{c})caligraphic_M ( None , bold_italic_c ) in section 3.4.

🔼 This figure shows the recall of the relevance scanning module at different cosine similarity thresholds. The relevance scanning module pre-filters rules, removing those deemed irrelevant to the image before further processing. The graph plots the recall (proportion of actually relevant rules correctly identified) against the cosine similarity threshold used for filtering. A higher threshold indicates stricter filtering, potentially missing more relevant rules but improving efficiency. The results demonstrate that the module effectively identifies most relevant rules while significantly reducing the number of rules that need subsequent processing.

read the caption

(a) Recall for ground truth rules.

🔼 The figure shows the fraction of safety rules remaining after applying a relevance scanning module to filter out rules irrelevant to the input image. The x-axis represents the cosine similarity threshold used for filtering, and the y-axis shows the percentage of rules retained after filtering. The graph illustrates the effectiveness of the module in reducing the number of rules processed while maintaining a high recall for ground-truth violated rules. A lower percentage of remaining rules indicates higher efficiency, as fewer rules need to be evaluated by the main model.

read the caption

(b) Fraction of remaining rules.

🔼 This figure demonstrates the effectiveness of the Relevance Scanning module (Section 3.2) in filtering out irrelevant rules before processing images for safety assessment. Using CLIP (Radford et al., 2021), the module achieves high recall (successfully keeping most of the ground-truth violated rules) while significantly reducing the number of rules to be processed in the next stage. The graph shows the trade-off between recall and the proportion of rules remaining after filtering, demonstrating the module’s ability to efficiently pre-process the rules.

read the caption

Figure 7: Detailed performance of Relevance Scanning module (see Section 3.2) with CLIP (Radford et al., 2021) on OS Bench. This module effectively filters out a significant proportion of irrelevant rules for the inspected images, while successfully retaining most of the ground-truth violated rules for forwarding to the next phase.

🔼 This figure visualizes the distribution of score differences obtained by comparing the token probability scores of whole images and their corresponding centric-region-removed versions. The image-level debiasing approach, detailed in Figure 4, aims to mitigate bias stemming from non-centric image content. The distribution reveals how effectively this approach distinguishes between situations where the presence of non-centric content significantly influences the score (large difference) versus those where it has minimal impact (small difference). The x-axis represents the calculated score difference, and the y-axis shows the percentage of images falling within each score difference range.

read the caption

Figure 8: Distribution of score differences calculated using our image-level debiasing approach (see Figure 4).

🔼 This figure shows the prompt template used to assess the objectivity of the safety guidelines. The prompt instructs an evaluator to act as an impartial judge, providing a short explanation of the guideline’s objectivity before rating its objectivity on a scale of 1 to 10 (10 being the most objective). This helps ensure that the rules used for image safety judgment are clear, unbiased, and easily understood by the MLLM.

read the caption

Figure 9: Prompt for measuring rule objectivenessb based on the template in Zheng et al. (2024).

🔼 This figure illustrates the two-stage process used to prepare data for the image safety judgment task. The first stage involves extracting preconditions from safety rules using a large language model (LLM). The provided examples show how complex safety rules are broken down into simpler, more manageable preconditions, making them easier for the MLLM to process in the subsequent judgment stage. The second stage focuses on extracting central object words from preconditions. This involves using the LLM again to identify the key objects or parts of objects described in the preconditions, ensuring efficient processing during image analysis. This helps to focus the model’s attention on the most relevant aspects of the image for safety determination, improving accuracy and efficiency.

read the caption

Figure 10: Detailed process for precondition extraction and central object word extraction.

🔼 This figure shows the recall of the relevance scanning module at different cosine similarity thresholds. The relevance scanning module filters out irrelevant rules before passing relevant ones to the MLLM for processing. A higher recall indicates that the module successfully retains most of the ground-truth violated rules, ensuring that important safety rules are not missed. The x-axis represents the cosine similarity threshold, and the y-axis represents the recall of ground-truth violated rules.

read the caption

(a) Recall for ground truth rules.

🔼 This table compares the performance of the proposed CLUE method against several zero-shot baselines on the OS Bench dataset. The zero-shot baselines use different strategies for determining image safety, such as directly answering ‘yes/no’ or using chain-of-thought reasoning. The models used for evaluation include several different MLLMs with varying parameter sizes and architectures. The table displays the recall, accuracy, and F1-score for each method and model combination, providing a comprehensive performance comparison on the image safety classification task.

read the caption

Table 2: Comparison to zero-shot baseline methods on distinguishing safe and unsafe images in OS Bench.

🔼 This table compares the performance of the proposed method (CLUE) against existing fine-tuning based methods for image safety assessment. Because the proposed method does not rely on human-labeled data, it’s compared to models pre-trained on different datasets and then evaluated on the same test set (OS Bench). The key takeaway is that fine-tuning approaches struggle to generalize beyond the specific safety rules they were trained on, highlighting the advantage of CLUE’s zero-shot capability.

read the caption

Table 3: Comparison to fine-tuning based baseline methods on distinguishing safe and unsafe images in OS Bench. Since our setting requires constructing the detector without human labeling, we compare our method to the default models trained on their respective datasets and inference on OS Bench. The key aim of this table is to show that existing fine-tuning-based methods lack generalizability beyond the safety rules used in training/fine-tuning.

🔼 This table presents the binary classification performance of the CLUE method (using the InternVL2-76B model) on distinguishing between images that violate specific safety rules and those that are borderline safe. The performance is measured using precision, recall, accuracy, and F1-score for each safety rule. The safety rules themselves are detailed in Table 1 of the paper. This evaluation shows how well the model identifies rule violations and separates them from near misses.

read the caption

Table 4: Binary classification performance of our method with InternVL2-76B on images violating each rule and the corresponding borderline-safe images. Detailed rules used are shown in Table 1.

🔼 This table compares the performance of a zero-shot image safety judgment model using two different sets of rules: an objectified set and the original set. The objectified rules were created by modifying the original rules to make them more objective and less ambiguous. The results show that the model achieves significantly higher accuracy when using the objectified rules, demonstrating the importance of objectifying safety rules for improving the performance of zero-shot image safety judgment models.

read the caption

Table 5: Results under objectified rule and the original one. The accuracy is much higher for the objectified rule, indicating constitution objectification is important for the zero-shot image safety judgment task.

🔼 This table presents an ablation study evaluating the impact of the debiased token probability-based judgment module on the overall performance of the CLUE model. It compares the accuracy and F1-score achieved with and without this module, highlighting its contribution to improving the model’s zero-shot image safety judgment capabilities.

read the caption

Table 6: Effects of debiased token probability based judgment.

🔼 This table presents the original set of safety rules used for image content moderation before the rules were modified to improve their objectivity for use with a multi-modal large language model (MLLM). Each rule is shown along with a score representing how objective the rule was considered to be before modification.

read the caption

Table 7: Original safety constitution before the objectification.

🔼 This table lists seed prompts used to generate images for the Objective Safety Bench (OS Bench) dataset. Seed prompts are initial prompts given to a text-to-image model to create both unsafe and safe images. For each of several rules defined for image safety, there are two types of prompts: those designed to elicit unsafe images violating the rule and those designed to elicit safe images that closely approach but do not violate the rule. The table is crucial because the OS Bench dataset was specifically created to address the lack of a benchmark dataset that uses objective rules for image safety assessment.

read the caption

Table 8: Detailed seed prompts used to construct OS Bench.

🔼 This table presents a detailed breakdown of the performance of various methods in distinguishing between unsafe images (violating specific rules) and borderline-safe images using the InternVL2-76B model. It shows precision, recall, accuracy, and F1-score for each rule defined in Table 1, offering a granular view of each model’s ability to correctly identify unsafe content while minimizing false positives.

read the caption

Table 9: Detailed binary classification performance of different methods with InternVL2-76B (Chen et al., 2023) on images violating each rule and the corresponding borderline-safe images. Detailed rules used are shown in Table 1.

🔼 This table presents the ablation study results focusing on the impact of precondition extraction on the model’s performance. It compares the model’s accuracy and F1-score with and without the precondition extraction module. The results demonstrate the importance of the precondition extraction module in enhancing the model’s ability to perform effective safety judgments.

read the caption

Table 10: Effects of Precondition Extraction.

🔼 This table presents an ablation study evaluating the impact of using score differences between whole images and their centric-region-removed counterparts on the model’s performance. It compares the recall and number of cascaded reasoning processes required for each image. The results demonstrate whether this method improves the recall and the efficiency of the overall approach.

read the caption

Table 11: Effects of score differences between whole and centric-region-removed images.

🔼 This table presents the average processing time taken by the proposed method, CLUE, to analyze a single image using various Multimodal Large Language Models (MLLMs). The runtime is measured for different models (InternVL2-8B-AWQ, LLaVA-v1.6-34B, InternVL2-76B) using specific inference backends (TurboMind, SGLang) and hardware configurations (Nvidia A100 GPUs). The information showcases the efficiency of the CLUE method across different scales of MLLMs and highlights the computational cost in real-world image safety judgment tasks.

read the caption

Table 12: Average time cost for our method on different MLLMs.