Granite Guardian

🔼 This figure shows the prompt used to generate benign prompts for the synthetic data used to train the Granite Guardian model. The prompt instructs a language model to create a list of requests that might initially seem harmful but are actually harmless. The goal is to help the language model learn to distinguish between genuinely harmful requests and those that are benign but might appear otherwise. It specifies the number of requests needed, a description of the type of safety, and examples to help guide the generation. The purpose is to provide contrastive examples to those that are harmful, aiding in the training of a more robust and less prone to false-positive model.

read the caption

Figure 2: Prompt for benign prompt generation

🔼 This figure shows the prompt used to generate synthetic data for typically harmful prompts. The prompt instructs an LLM to generate diverse requests that a language model should refuse, even if they might initially seem harmless. The prompt requests that each request be on a new line and prefixed with “Request:”. Specific parameters such as subcategory, fine-grained subcategory and description of the request can be filled to obtain different types of harmful requests. This methodology allows for the generation of a large dataset of harmful prompts that can be used to train a safety model.

read the caption

Figure 3: Prompt for typical harmful prompt generation

🔼 This figure shows a prompt designed to generate adversarial harmful prompts. The prompt instructs an AI to act as a red-teaming assistant, tasked with revising simple harmful prompts into more sophisticated and harder-to-detect adversarial versions. This involves using multiple revision strategies to make the harmful intent more subtle and less likely to be flagged by safety mechanisms. The prompt provides a simple harmful request as input, along with a set of revision strategies, each defined with its description and examples. The AI is expected to produce a revised adversarial prompt that is logically coherent, maintains the same harmful intent, but is harder to detect using common safety measures.

read the caption

Figure 4: Prompt for adversarial harmful prompt generation

🔼 This figure presents an example of a revision strategy used to transform a simple harmful prompt into a more sophisticated adversarial prompt. The goal is to create prompts that can bypass the safety mechanisms of language models and elicit harmful responses. The example shows how a malicious prompt can be broken down into smaller, seemingly innocuous parts (payloads) that would not individually trigger a safety alert but can be reassembled by a language model to produce harmful content. This technique highlights the importance of considering more complex, adversarial prompt construction during safety evaluations of language models. The figure is crucial for illustrating the types of adversarial techniques that language models are vulnerable to, and how these need to be accounted for in safe deployment strategies.

read the caption

Figure 5: Example revision strategy for adversarial prompt transformation

🔼 This figure shows a prompt designed for generating synthetic data to evaluate RAG (Retrieval Augmented Generation) models’ performance on hallucination-related risks. The prompt instructs the model to generate several different types of answers to a given question and short answer, each designed to test a specific aspect of RAG quality. These include correct answers, answers that are irrelevant to the question, incorrect answers taken from the document, and questions that themselves are not relevant to the provided context. This process helps create a comprehensive and diverse dataset for training and evaluating RAG models’ ability to produce accurate and contextually relevant responses.

read the caption

Figure 6: Prompt for RAG synthetic data generation

🔼 This table presents the demographic information of the annotators who participated in the data labeling process for the Granite Guardian project. It shows the annotators’ birth year, age, gender, education level, ethnicity, and region. This data provides valuable context on the diversity of the individuals involved in creating the dataset and can help understand any potential biases in the dataset.

read the caption

Table 2: Annotator Demographics

🔼 This table displays the level of agreement between multiple human annotators who independently labeled prompts and responses for various risk categories. It quantifies the inter-annotator reliability for each risk category (Bias, Jailbreaking, Violence, Profanity, Sexual Content, Unethical Behavior, AI Refusal, and Other) by providing the inter-annotator agreement score (presumably Cohen’s Kappa or a similar metric) separately for prompts and responses. Higher scores indicate stronger agreement among annotators, suggesting higher quality and consistency in the labeling process.

read the caption

Table 3: Inter-annotator agreement for prompt/response labels

🔼 This table presents the details of eight public benchmark datasets used to evaluate the performance of the Granite Guardian model in detecting harmful content in both prompts and responses. It lists each dataset’s name, the source reference, the total number of samples, the number of samples classified as benign, the number of samples classified as harmful, and the type of data (prompt or response). Note that some datasets have undergone sub-sampling (*), refusal responses flagged as benign (†), and compliance responses flagged as harmful(‡). This information is crucial in understanding the evaluation methodology and the nature of the data used to assess the model’s performance.

read the caption

Table 4: Details of the public benchmarks used for evaluation. ∗ indicates sub-sampling from the original set, †refers to refusal responses flagged as benign, and ‡refers to compliance responses flagged as harmful.

🔼 This table lists the datasets used to evaluate the performance of RAG (Retrieval Augmented Generation) models on the task of groundedness. It details the name of each dataset, the reference where it is described, the total number of samples, and the breakdown of those samples into consistent and inconsistent examples. Each dataset focuses on different NLP tasks like summarization, dialogue, and paraphrasing, providing a comprehensive evaluation of RAG across various scenarios.

read the caption

Table 5: Details of the TRUE benchmarks used for RAG evaluation.

🔼 This table presents a comparison of the performance of Granite Guardian’s harm risk detection model against Llama Guard and ShieldGemma models. The comparison uses aggregated datasets for harmful content detection, focusing on the umbrella harm risk definition. The table shows several evaluation metrics (AUC, AUPRC, F1, Recall, and Precision). The baselines (Llama Guard and ShieldGemma) were adapted to ensure a fair comparison. The best and second-best performances for each metric are highlighted in bold and underlined, respectively. This allows readers to easily identify the relative strengths of each model in detecting harmful content.

read the caption

Table 6: Results on aggregated datasets for harmful content detection comparing Granite Guardian (using the umbrella harm risk definition) with Llama Guard and ShieldGemma model families. Baselines are suitably adapted for direct comparison (see section 6.1 for details). Numbers in bold represent the best performance within a column, while underlined numbers indicate the second-best.

🔼 This table presents the performance of various models, including Granite Guardian, in detecting harmful content in both prompts and responses. The models are evaluated across multiple datasets, categorized by whether the harm is present in the prompt or the response. The F1 score and Area Under the ROC Curve (AUC) are reported as key performance metrics. The baselines are modified to facilitate fair comparisons with Granite Guardian. Numbers in bold represent the best performance in each column, and underlined numbers indicate the second-best performance.

read the caption

Table 7: F1/AUC results across different datasets, categorised across prompt harmfulness and response harmfulness. Baselines are suitably adapted for direct comparison (see section 6.1 for details). Numbers in bold represent the best performance within a column, while underlined numbers indicate the second-best.

🔼 This table presents the Area Under the Curve (AUC) scores for different models on the TRUE benchmark dataset, specifically focusing on the groundedness aspect of RAG (Retrieval Augmented Generation). The TRUE dataset encompasses various natural language processing (NLP) tasks designed to assess the factual consistency of generated text. The models’ AUC scores are compared across multiple sub-datasets within TRUE, allowing for a comprehensive evaluation of their performance in accurately identifying grounded vs. hallucinated statements. The bold numbers in each column highlight the top-performing model for that specific task, while underlined numbers denote the second-best performance. This enables easy comparison of the models’ strengths and weaknesses across a diverse range of NLP tasks.

read the caption

Table 8: AUC results on the TRUE dataset for groundedness. Numbers in bold represent the best performance within a column, while underlined numbers indicate the second-best.

🔼 This table provides a comprehensive list of risk categories and their detailed definitions as used in the Granite Guardian model. It covers a wide range of risks, categorized into general harm, social biases, profanity, sexual content, unethical behavior, violence, jailbreaking, and RAG (Retrieval-Augmented Generation)-specific risks such as context relevance, groundedness, and answer relevance. Each risk category includes a clear and concise definition to facilitate a thorough understanding of the model’s capabilities and limitations in risk detection.

read the caption

Table 9: Risk Definitions

🔼 This table details how the safety instruction template is used for different risk categories. The ‘Risk Type’ column lists the various risk types, such as harmful content (Harm++), jailbreaking, and different aspects of Retrieval Augmented Generation (RAG) quality (context relevance, groundedness, and answer relevance). The ‘Secondary’ and ‘Primary’ columns indicate which parts of the prompt and response text (user or assistant) the safety agent focuses on when assessing the risk. This is crucial to understanding how the model processes the information for different kinds of risks and enables flexibility in adapting the model to various safety concerns.

read the caption

Table 10: Designated roles in the safety instruction template for different risk categories. Harm++ refers to all harmful content risks (Section 2.1.1). The “Primary” column indicates the tag that determines the safety agent’s focus, while the “Secondary” column, in conjunction with the “Primary” tag, specifies the content to be included in the safety instruction template, as detailed in Section 4.1.

🔼 This table presents the Area Under the Curve (AUC) and True Positive Rate (TPR) at various False Positive Rate (FPR) thresholds (0.1, 0.01, and 0.001) for several models. The Area Under the Curve (AUC) provides an overall measure of the model’s ability to distinguish between classes. The True Positive Rate (TPR), at a fixed FPR, measures the model’s effectiveness at correctly identifying true positives, while controlling the rate of false positives. This is crucial for applications with stringent requirements for minimizing false positives. The models compared include Granite Guardian (two versions: 2B and 8B), Llama Guard models (multiple versions), and ShieldGemma models (multiple versions). The table highlights the best performing model for each metric at each threshold using bold font and the second-best performing model with underlined font.

read the caption

Table 11: AUC and TPr results on specific FPr thresholds (i.e., with FPr equal to 0.1, 0.01, 0.001). Numbers in bold represent the best performance within a column, while underlined numbers indicate the second-best.