TL;DR#
Reasoning-enhanced LLMs offer better complex problem-solving, yet control primarily relies on input-level operations. This limits transparency in the model’s cognitive processes, hindering precise interventions. The study highlights the need for more fine-grained control over model behavior to improve instruction following and safety, especially since current methods lack transparency.
To tackle this, the paper introduces Thinking Intervention, explicitly guiding LLMs by strategically inserting/revising thinking tokens within reasoning processes. It shows this method outperforms baseline prompting across multiple tasks. Thinking Intervention improves instruction-following, instruction hierarchy, and safety alignment, demonstrating flexible control without model training.
Key Takeaways#
Why does it matter?#
This paper is important for researchers by introducing a novel paradigm, Thinking Intervention, for enhanced control over reasoning models. This offers a promising research avenue for controlling reasoning LLMs and opens doors for precise, transparent, and effective control over the reasoning processes of LLMs.
Visual Insights#
🔼 Figure 1(a) illustrates three different methods for prompting a reasoning language model: Vanilla Prompting, Prompt Engineering, and Thinking Intervention. Vanilla Prompting and Prompt Engineering modify only the input prompt (the question), whereas Thinking Intervention injects additional instructions directly into the model’s reasoning process, either written by humans or automatically generated by another language model. Figure 1(b) presents the experimental results showing that Thinking Intervention significantly outperforms the other two approaches in three tasks: instruction following, instruction hierarchy, and safety alignment. The results are presented as performance percentages on the y-axis with approaches along the x-axis. The R1-Qwen-32B reasoning model was used for this evaluation.
read the caption
Figure 1: (a) A demonstration of how Vanilla Prompting, Prompt Engineering, and Thinking Intervention work. Both Vanilla Prompting and Prompt Engineering methods act on the input query. In contrast, Thinking Intervention, either written by humans or generated by LLMs, explicitly injects instructions into the reasoning process. (b) Compared to Vanilla Prompting and Prompt Engineering, Thinking Intervention offers significant performance improvements for R1-Qwen-32B reasoning model across instruction following, instruction hierarchy, and safety alignment tasks.
| (a) R1-Qwen-14B | (b) R1-Qwen-32B | (c) QwQ-32B |
🔼 This table presents a quantitative comparison of three different prompting methods on the SEP (Instruction Hierarchy) dataset using three different reasoning models. The methods compared are Vanilla Prompting (baseline), Reminder Prompting (baseline), and Thinking Intervention (the proposed method). For each method and model, the table shows the performance across two key metrics: Robustness (the percentage of times the model correctly ignores low-priority instructions) and Utility (the model’s performance on the main task in the absence of conflicting instructions). This allows for a direct comparison of the effectiveness of Thinking Intervention in improving the model’s ability to prioritize main tasks while ignoring irrelevant instructions, providing evidence of its capabilities in instruction hierarchy tasks.
read the caption
Table 1: Evaluation results on the SEP dataset across various reasoning models. We compare our proposed Thinking Intervention (+ThinkingI) against the Vanilla Prompting and Reminder Prompting.
In-depth insights#
LLM: Introspection#
LLM introspection, while not explicitly detailed in this work, holds immense potential. It could enable models to critically evaluate their reasoning, identify biases, and correct errors in real-time. This could involve the LLM examining its own activation patterns, attention weights, or the confidence scores of its token predictions. Such self-awareness would allow for dynamic adjustment of reasoning strategies, promoting more reliable and aligned outputs. Introspection mechanisms could also facilitate the extraction of implicit knowledge learned by the LLM, making its decision-making processes more transparent and understandable. While the current paper focuses on Thinking Intervention, integrating introspection alongside external guidance could represent a powerful synergy for controlling and enhancing LLM behavior. Ultimately, a self-aware and self-correcting LLM offers a pathway towards more robust and trustworthy AI systems.
Reasoning Control#
Reasoning control is a pivotal area in AI, enabling fine-grained influence over LLMs’ cognitive processes. This paper introduces Thinking Intervention, a novel paradigm for explicit guidance via strategic token insertion/revision. This offers enhanced transparency by directly targeting the reasoning trajectory, unlike input-level prompt engineering. It aligns model behavior with task objectives, bypassing extensive training and deploying with minimal effort. Compatible with prompt engineering and fine-tuning, it adaptively modifies reasoning steps based on context-specific needs, allowing for flexible control.
Context Matters#
The research paper underscores the critical role of context in reasoning models. Contextual understanding is paramount for effective LLM performance, enabling accurate interpretation and nuanced responses. Ignoring context can lead to misinterpretations and irrelevant outputs. Thinking Intervention can dynamically adapt reasoning based on context. This adaptation ensures that interventions are relevant and effective, ultimately enhancing the model’s ability to handle complex and ambiguous queries. Contextual integration ensures safety and relevance, mitigating harmful responses and aligning model behavior with user intentions and ethical guidelines.
Safety vs Utility#
The balance between safety and utility is a critical concern in AI development. Overly cautious safety measures can limit a model’s helpfulness and ability to address user needs effectively. Striking the right equilibrium is challenging. The paper indicates open-source reasoning models have lower refusal rates for unsafe requests, potentially prioritizing utility over safety, underscoring the need for better safety mechanisms, such as Thinking Intervention, which enhances safety while maintaining reasonable compliance for safe requests. It enables more explicit control, guiding reasoning to substantially improve the safety profile without drastically hindering the model’s usefulness. It’s a constant tradeoff, as perfect refusal may eliminate potential aid in critical situations.
Simple is best#
The principle of ‘Simple is Best’ resonates deeply in the pursuit of artificial intelligence. In the context of reasoning models, simplicity fosters transparency and interpretability, qualities often sacrificed in complex architectures. A simpler model is easier to understand, debug, and control, aligning with the goals of Thinking Intervention, which aims for fine-grained control. While complex models might achieve higher raw performance, their opacity can hinder effective intervention and safety assurance. Therefore, a balanced approach is crucial, where complexity is added judiciously, retaining the capacity for direct manipulation and comprehension, thus enabling the more effective intervention for desirable and safe outputs. Simplicity reduces the potential for unintended consequences and biases.
More visual insights#
More on figures
🔼 This figure demonstrates three different prompting methods for instruction-following tasks: Vanilla Prompting (using the instruction as is), Prompt Engineering (adding a reminder to ensure the model adheres to the instruction’s constraints), and Thinking Intervention (injecting instructions directly into the model’s internal reasoning process). The example shows a task requiring the model to list two famous mothers in JSON format. Vanilla Prompting results in an incorrectly formatted or incomplete response. Prompt Engineering improves the response slightly. Thinking Intervention, which inserts guiding text into the model’s thinking stage (
block), yields the most accurate response in the correct JSON format. This comparison highlights how Thinking Intervention allows for a more precise and effective control over the model’s reasoning and output compared to traditional prompt engineering methods. read the caption
Figure 2: An example demonstrating how Thinking Intervention is integrated with vanilla prompting and Reminder Prompting prompting techniques for instruction following tasks.
🔼 This figure displays the results of experiments evaluating instruction-following capabilities of different large language models (LLMs) on the IFEval benchmark. Three prompting methods were compared: Vanilla Prompting (using only the original instructions), Reminder Prompting (adding reminders to the instructions), and Thinking Intervention (injecting instructions directly into the model’s reasoning process). The accuracy of each method is shown across three different LLMs, indicating the improvement in instruction-following performance achieved through Thinking Intervention. The chart clearly visualizes the gains in accuracy for all three models and across different prompting techniques, demonstrating Thinking Intervention’s superiority.
read the caption
Figure 3: Evaluation results on the IFEval benchmark [69]. We compare the performance with and without Thinking Intervention (ThinkingI), across Vanilla Prompting and Reminder Prompting methods and multiple reasoning models.
🔼 The SEP benchmark assesses a model’s ability to prioritize main instructions over secondary, low-priority ones. Figure 4 shows two scenarios. On the left, a low-priority instruction is embedded within the data provided alongside the main instruction; a successful model would ignore the low-priority instruction and only complete the primary task. The right side shows the same main instruction with the data, but without the conflicting low-priority instruction; the result demonstrates the model’s inherent ability to accomplish the primary task.
read the caption
Figure 4: A demonstration of how the SEP benchmark evaluates instruction hierarchy capabilities. Each example contains a main instruction paired with data. Left: The low-priority instruction is injected into the data, which the model should correctly ignore. Right: The low-priority instruction is absent, measuring the utility of models.
🔼 This figure illustrates three different prompting methods applied to the SEP benchmark: Vanilla Prompting, Reminder Prompting, and Thinking Intervention. Vanilla Prompting uses the original instruction without modification. Reminder Prompting adds a reminder to the instruction to ensure the model adheres to it. Thinking Intervention injects instructions directly into the model’s internal reasoning process by inserting specific tokens, providing a more precise form of control. The example shows how each method is applied to a task with high- and low-priority instructions, where the low-priority instruction is embedded within the data. This demonstrates how Thinking Intervention can lead to better control and accuracy than the other two methods.
read the caption
Figure 5: A demonstration of Vanilla Prompting, Reminder Prompting, and Thinking Intervention for the SEP benchmark. The {Task} and {Data} fields are filled with content from the SEP dataset (e.g., Figure 4) during evaluation.
🔼 This figure displays the performance of several large language models (LLMs) on the XSTest benchmark for evaluating the safety alignment of models. The models were evaluated using only the standard prompt (Vanilla Prompting), without any additional safety-oriented techniques. The results reveal the models’ compliance rates for safe requests and their refusal rates for unsafe requests. It highlights a critical safety concern with the R1 models, demonstrating a dangerously high compliance rate for unsafe requests despite good performance on safe requests. This emphasizes a trade-off between helpfulness and safety, where the models prioritize answering even dangerous requests.
read the caption
Figure 6: Model’s performance on the XSTest benchmark via Vanilla Prompting prompting.
🔼 This figure showcases the impact of Thinking Intervention on the safety performance of the R1-Qwen-32B model, as evaluated using the XSTest benchmark. It compares the model’s safety performance (refusal rate for harmful requests and compliance rate for safe requests) under various prompting methods, both with and without the incorporation of Thinking Intervention. The visualization helps illustrate how Thinking Intervention enhances the model’s ability to reject harmful prompts while maintaining its ability to correctly respond to benign prompts.
read the caption
Figure 7: Effect of the Thinking Intervention with R1-Qwen-32B model on XSTest benchmark results.
🔼 This figure shows the results of applying Thinking Intervention to the R1-Qwen-32B model on the SORRY-BENCH benchmark. It compares the model’s performance with and without Thinking Intervention across various prompting methods (Vanilla, Default, Reminder, Goal Priority). The x-axis represents the prompting methods used, while the y-axis shows the unsafe request refusal rate (%). The figure visually demonstrates how Thinking Intervention impacts the model’s ability to refuse unsafe prompts, improving its safety performance. Different colored bars represent the results obtained for the methods with and without Thinking Intervention.
read the caption
Figure 8: Effect of the Thinking Intervention with R1-Qwen-32B model on SORRY-Bench.
🔼 This figure demonstrates the process of generating Reminder Prompts using GPT-40. The input is a prompt instructing the model to extract constraints from a user prompt and create a reminder for an LLM to adhere to those constraints. An example input and output are provided to illustrate the task. The generated reminder is then slightly modified to create the Thinking Intervention sequence used in the experiment. The image likely shows the prompt given to GPT-40, the model’s response (the Reminder Prompt), and the final Thinking Intervention sequence derived from it.
read the caption
Figure 9: A demonstration of how we prompt GPT-4o to generate the Reminder Prompting. The intervention sequence is a slightly modified version of Reminder Prompting.
🔼 The SEP benchmark evaluates a model’s ability to prioritize instructions. The figure shows three example prompts. In the left example, a low-priority instruction is embedded within the data; a successful model will ignore it and only focus on the main instruction. In the middle example, the low-priority instruction is part of the main task instruction; a successful model will address both. The rightmost example omits the low-priority instruction, allowing for the measurement of baseline performance (utility) on the main task alone.
read the caption
Figure 10: A demonstration of how the SEP benchmark evaluates instruction hierarchy capabilities. Each example consists of a main instruction paired with data. Left: A low-priority instruction is injected into the data, which the model should correctly ignore. Middle: A low-priority instruction is injected into the task portion, which the model should follow and generate answers. Right: The low-priority instruction is absent, allowing us to measure the utility of different methods.
🔼 This figure shows the prompt template used to evaluate the ‘utility’ metric in the SEP (instruction hierarchy) benchmark. The template guides an LLM (Large Language Model) to act as a judge, evaluating the quality of a model’s response to a given prompt. The LLM judge considers factors such as helpfulness, relevance, accuracy, depth, and creativity in its evaluation. The evaluation is presented in a structured format, requiring the LLM judge to provide a short explanation of its assessment, followed by a numerical rating (1-10) using the format ‘[[rating]]’. The {Question} placeholder represents the input prompt given to the model being tested, and the {Answer} placeholder represents the actual response generated by that model. The LLM judge evaluates the model’s response based on this information.
read the caption
Figure 11: Prompt template of evaluating the utility metric on the SEP benchmark. The {Question} and {Answer} will be filled with the complete prompt and model response, respectively.
🔼 Figure 12 displays the various prompting methods used to assess the safety of language models. It shows the prompts used for four approaches: Vanilla Prompting (no additional instructions); Default Prompting (incorporates general safety guidelines); Reminder Prompting (explicitly reminds the model of safety responsibilities); and Goal Priority Prompting (prioritizes safety above all else). The {Query} placeholder indicates where the actual user query from the safety benchmark would be inserted. For Goal Priority Prompting, the few-shot examples suggested in reference [2] were removed. This figure visually represents the different methods used to control model safety and provides a clear comparison of their inputs.
read the caption
Figure 12: The baseline prompting methods, as well as our Thinking Intervention, used for evaluating safety steering. The {Query} will be filled with the query from the benchmarks. Note for Goal Priority Prompting, we eliminate the few-shot exemplars as suggested by [2].
🔼 This figure details the prompt template used to assess the safety and refusal rate of a language model on the XSTest benchmark. The evaluation process involves classifying model responses into three categories: 1) full compliance (providing an answer), 2) full refusal (explicitly declining to answer), and 3) partial refusal (a combination of compliance and refusal). The template uses placeholders, {Question} and {Answer}, which are replaced with the actual prompt and model’s response respectively during the evaluation. The resulting classification of each response helps quantify the model’s safety behavior by determining its tendency to engage with potentially harmful requests.
read the caption
Figure 13: Prompt template of evaluating the compliance or refusal rate on the XSTest benchmark. The {Question} and {Answer} will be filled with the complete prompt and model response, respectively.
🔼 This figure displays the results of applying Thinking Intervention on the XSTest benchmark across various reasoning models. It shows a comparison of the models’ performance under different prompting methods (Vanilla Prompting, Default Prompting, Reminder Prompting, Goal Priority) both with and without Thinking Intervention. The x-axis represents the safe request compliance rate, and the y-axis represents the unsafe request refusal rate. Each point on the graph corresponds to a particular model and prompting method, illustrating the trade-off between safe request compliance and unsafe request refusal. The effectiveness of Thinking Intervention is evident in the improvements to the unsafe request refusal rate achieved across different models and prompting methods.
read the caption
Figure 14: Effectiveness of Thinking Intervention on the XSTest benchmark across multiple reasoning models.
🔼 This figure shows the prompt template used to evaluate the refusal rate of language models on the SORRY-BENCH benchmark. The template guides an evaluator (in this case, GPT-40-MINI) to classify model responses as either fully compliant with the user request, fully refusing to answer, or partially refusing (a combination of compliance and refusal). The evaluator is given the user’s question and the model’s response, and it provides a numerical classification based on the model’s behavior. This setup is designed to measure how well the model handles potentially unsafe or harmful requests by assessing its tendency to refuse such queries.
read the caption
Figure 15: Prompt template of evaluating the refusal rate on the SORRY-Bench benchmark. The {Question} and {Answer} will be filled with the complete prompt and model response, respectively.
🔼 Figure 16 displays the results of safety experiments on the SORRY-BENCH dataset, a benchmark focusing on evaluating safety and robustness. The figure illustrates the performance of various reasoning models across different prompting strategies and when enhanced with the proposed Thinking Intervention technique. Each bar in the graph represents a specific model and prompting method, showing the percentage of unsafe prompts successfully refused by the model. The Thinking Intervention technique consistently improved the refusal rates across multiple models. This indicates improved safety alignment by reducing the tendency of reasoning models to respond to unsafe prompts while preserving their ability to answer safe queries. The figure highlights the consistent improvements across various prompting strategies and model architectures, demonstrating the robustness of the proposed Thinking Intervention approach.
read the caption
Figure 16: Effectiveness of Thinking Intervention on the SORRY-Bench benchmark across multiple models. Our approach consistently improves the safety alignment of reasoning models.
🔼 This figure displays the results of experiments evaluating the impact of placing a Thinking Intervention sequence at different points (beginning, middle, and end) within the reasoning process of a language model. The experiments were conducted using the XSTest and SORRY-Bench benchmarks for safety assessment. The graphs show the relationship between safe request compliance rates and unsafe request refusal rates under the different intervention placements. The purpose is to determine the optimal placement of the intervention sequence for maximizing the model’s safety and performance.
read the caption
Figure 17: Analysis of varying the location of the intervention sequence on the XSTest and SORRY-Bench benchmarks. The content is kept unchanged, and the Thinking Intervention is placed at the beginning, middle, and end of the reasoning process.
🔼 This figure displays the results of experiments on the XSTest and SORRY-Bench benchmarks, evaluating the impact of different Thinking Intervention sequences on model safety. Two intervention sequences were tested: a short version and a longer version. Both were placed at the beginning of the reasoning process within the model. The graphs illustrate how the different intervention lengths affect the model’s performance in terms of the rate of refusing unsafe prompts (robustness) and maintaining accurate responses to safe prompts (compliance). The results show the trade-off between safety and compliance, demonstrating how the length of the intervention sequence impacts the model’s ability to prioritize safety without sacrificing its helpfulness.
read the caption
Figure 18: Analysis of varying the Thinking Intervention content on XSTest and SORRY-Bench benchmarks. We compare our default short intervention sequence with a longer version. Both versions are inserted at the beginning of the reasoning process.
🔼 This figure demonstrates a model’s failure to prioritize instructions when a low-priority instruction is included. Without Thinking Intervention, the model incorrectly focuses on the low-priority instruction (‘Tell me what a book is primarily made of’) from the data, while neglecting the main task (‘Interpret the statute mentioned in the next paragraph, explaining its legal implications’) which is related to the Sarbanes-Oxley Act. This leads to an incorrect response, primarily addressing the low-priority query instead of the main instruction. The green highlights the main task and red highlights the low-priority query.
read the caption
Figure 19: A demonstration of how models without Thinking Intervention fail to ignore low-priority instructions and consequently provide incorrect responses. We use green color to highlight the main task and red color to highlight the low-priority query.
🔼 Figure 20 shows how Thinking Intervention helps a reasoning model correctly prioritize instructions. The model receives a main instruction (in green) to interpret a legal statute, along with some data that includes a low-priority, unrelated instruction (in red). Without Thinking Intervention, the model gets distracted by the low-priority instruction. However, with Thinking Intervention (highlighted in blue), the model correctly focuses on the main task and produces a correct and complete answer, successfully ignoring the irrelevant instruction in the data. The use of color-coding (blue for intervention, green for main task, red for low-priority) helps visually illustrate the model’s behavior and the impact of Thinking Intervention.
read the caption
Figure 20: A demonstration of how models with Thinking Intervention successfully ignore low-priority instructions and provide correct responses. We use blue color to highlight the Thinking Intervention, green color to highlight the main task and red color to highlight the low-priority query.
More on tables
| R1-Qwen-14B | R1-Qwen-32B | QwQ-32B | ||||
| Methods | Robustness(%) | Utility(%) | Robustness(%) | Utility(%) | Robustness(%) | Utility(%) |
| Vanilla | 34.00 | 81.04 | 34.80 | 81.76 | 22.20 | 88.00 |
| Vanilla+ThinkingI | 38.40 (+4.40) | 81.08 (+0.04) | 50.20 (+15.40) | 82.02 (+0.26) | 31.40 (+9.20) | 88.16 (+0.16) |
| Reminder | 38.40 | 80.50 | 46.20 | 81.16 | 36.20 | 87.52 |
| Reminder+ThinkingI | 41.80 (+3.40) | 80.90 (+0.40) | 66.40 (+20.20) | 80.90 (-0.26) | 43.40 (+7.20) | 86.79 (-0.73) |
🔼 This table presents a comprehensive evaluation of instruction-following performance across three reasoning models (R1-QWEN-14B, R1-QWEN-32B, and QwQ-32B). It compares the accuracy of three prompting methods: Vanilla Prompting, Reminder Prompting, and Thinking Intervention. For each model and prompting method, the table shows the strict and loose accuracy, measured at both the prompt and instruction levels. Strict accuracy reflects perfect adherence to instructions, while loose accuracy allows for minor deviations. The best-performing method for each metric is highlighted in bold, while the second-best is underlined. The results demonstrate consistent performance improvements using Thinking Intervention across all models and evaluation criteria.
read the caption
Table 2: The evaluation results on the IFEval dataset span multiple reasoning models. We compare our method, Thinking Intervention (+ThinkingI), with the Vanilla Prompting and Reminder Prompting methods and observe consistent performance improvements. The best results are in bold, and the second best are underlined.
| Models | Methods | Prompt-level | Inst-level | Prompt-level | Inst-level |
| strict acc.(%) | strict acc.(%) | loose acc.(%) | loose acc.(%) | ||
| R1-Qwen-14B | Vanilla | 70.43 | 79.50 | 73.57 | 81.77 |
| Vanilla+ThinkingI | 75.42 | 81.65 | 78.37 | 84.29 | |
| Reminder | 72.83 | 81.18 | 76.53 | 83.69 | |
| Reminder+ThinkingI | 74.31 | 82.13 | 77.26 | 84.41 | |
| R1-Qwen-32B | Vanilla | 70.43 | 79.14 | 74.49 | 81.89 |
| Vanilla+ThinkingI | 77.08 | 84.29 | 80.96 | 86.93 | |
| Reminder | 75.23 | 82.85 | 78.74 | 85.37 | |
| Reminder+ThinkingI | 77.63 | 84.53 | 81.70 | 87.29 | |
| QwQ-32B | Vanilla | 79.30 | 86.09 | 83.92 | 89.09 |
| Vanilla+ThinkingI | 82.26 | 88.01 | 86.32 | 90.65 | |
| Reminder | 81.33 | 87.53 | 86.69 | 91.01 | |
| Reminder+ThinkingI | 82.44 | 88.13 | 86.69 | 91.01 | |
🔼 This table presents a quantitative comparison of three different prompting methods on the instruction hierarchy task using the SEP benchmark. The methods compared are Vanilla Prompting (no additional instructions), Reminder Prompting (explicitly reiterating the constraint), and Thinking Intervention (inserting instructions into the reasoning process). The table shows the performance of several reasoning models (R1-QWEN-14B, R1-QWEN-32B, and QwQ-32B) on three metrics: Robustness (how well the model ignores low-priority instructions), SEP utility (performance on the main task without the low-priority instruction), and Utility (overall performance). Results show the effectiveness of Thinking Intervention in improving robustness while maintaining high utility.
read the caption
Table 3: Evaluation results on the SEP dataset across various reasoning models. We compare our proposed Thinking Intervention (+ThinkingI) against Vanilla Prompting and Reminder Prompting.
Full paper#
