SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models

🔼 This table presents the performance comparison of different question answering methods on three benchmark datasets: TriviaQA, HotpotQA, and ASQA. Results are shown for three different large language models: Llama-3.2 3B, Llama-3.1 8B, and GPT-40. Each row shows the performance of a single model on a specific dataset using a different technique. The techniques include a baseline approach, Retrieval Augmented Generation (RAG), Chain-of-Thought (CoT) prompting, rephrase-and-respond (RaR), and the proposed SQUARE method. The best performing method for each model and dataset is highlighted in bold.

read the caption

Table 2: The main results of our experimentation. Each row group corresponds to the results for the given dataset, with each row showcasing the metric results for each model. The columns include all the main approaches, with bold highlighting the best result across all approaches.

🔼 This table compares three methods for aggregating the results of SQUARE: using no aggregation, summarizing the answers, and voting on the answers. It also shows how the performance of the Llama-3.1 8B model changes when varying the number of sub-questions (N) generated by the SQUARE method. The results are based on experiments using a few-shot learning approach with adapted examples, as detailed in Appendix A.2 of the paper.

read the caption

Table 3: Comparison of two aggregation methods in addition to SQuARE, and the effect of varying the number of sub-questions (N𝑁Nitalic_N). Results showcase the Llama-3.1 8B model with few-shot examples adapted for each approach, as detailed in Section A.2.

🔼 This table presents a comprehensive summary of the experimental results obtained across three different question-answering datasets (TriviaQA, HotpotQA, and ASQA) using three distinct language models (Llama-3.2 3B, Llama-3.1 8B, and GPT-40). For each dataset-model combination, the table compares the performance of seven different configurations: a baseline, RAG (Retrieval-Augmented Generation), CoT (Chain-of-Thought), RaR (Rephrase and Respond), and three variations of the SQUARE method with different numbers of generated questions (N=3, N=5, and N=10). Where applicable, both zero-shot (no prior examples) and few-shot (with example queries) results are reported, offering a detailed comparison of various prompting techniques on different model sizes and datasets.

read the caption

Table 4: Summary of results; comparing seven configurations on three datasets, using three models. When applicable, zero-shot/few-shot results are shown.

🔼 This table presents a verbatim example of an incorrect answer generated by a Llama-3.1 8B model using the chain-of-thought (CoT) prompting technique. The question is from the HotpotQA dataset. The correct answer, ‘March and April’, is present within the model’s reasoning steps. However, the model’s final answer is incorrect (‘Open City’). The discrepancy highlights the challenge of reliably extracting correct answers from the model’s intermediate reasoning steps and underscores the need for improved answer extraction methods.

read the caption

Table 5: Generated answer (verbatim) for a question from the HotpotQA dataset, produced by Llama-3.1 8B model, using the CoT technique with two few-shot examples. Correct answer is March and April, which is found in the reasoning chain, probably due to this fact appearing in the relevant external context provided to the model (not shown here). The final model’s answer is wrong, necessitating the need to correctly extract answers for better analysis and research.

🔼 Table 6 presents two example question-answer pairs used in the few-shot learning setting of the SQUARE method. These examples are provided to the model before it receives the actual question, to give it context and guide its reasoning. The SQUARE prompt is used, with a parameter ‘N’ set to 3, meaning the model generates and answers three intermediate questions before arriving at the final answer to the main query.

read the caption

Table 6: The two examples used in our few-shot configurations. Here, the SQuARE prompt is used, with N=3𝑁3N=3italic_N = 3.

🔼 Table 7 details the prompt used in the SQUARE experiment for answer summarization. This variant of the SQUARE method modifies the original prompt to instruct the LLM to not only generate and answer multiple sub-questions, but also to concisely summarize the information gleaned from those answers before arriving at a final answer to the original question.

read the caption

Table 7: Prompt used for the SQuARE supplement aggregation method, using answer summarization; see Section 4.1.

🔼 This table presents two example question-answer pairs used in an ablation study. The study examines the effect of adding a summarization step to the SQUARE method before the model provides a final answer. The examples illustrate how the SQUARE method, enhanced with summarization, works on two different question types. These examples are used as few-shot prompts within the broader experimental setup described in Section 4.1 of the paper.

read the caption

Table 8: Two-shot examples used in an ablation study where SQuARE is supplemented by an instruction to summarize the model’s learning before providing the final answer, see Section 4.1.

🔼 This table details the prompt used for the SQUARE (Sequential Question Answering Reasoning Engine) method’s supplement aggregation technique, specifically employing a majority voting approach to determine the final answer. The SQUARE method involves generating several sub-questions related to the main question and then using the answers to those sub-questions to arrive at a final answer. This particular variant uses a majority voting approach amongst the sub-questions’ answers to determine the final answer. This is contrasted with other aggregation methods explored in the paper (summarization and voting).

read the caption

Table 9: Prompt used for the SQuARE supplement aggregation method, using majority voting; see Section 4.1.

🔼 This table presents two examples used in an ablation study to evaluate a modified version of the SQuARE prompting technique. The modification involves instructing the language model to select the final answer by majority vote among the answers generated to several sub-questions. This table is part of an analysis to determine the effect of different answer aggregation methods on the model’s performance.

read the caption

Table 10: Two-shot examples used in an ablation study where SQuARE is supplemented by an instruction to pick the final answer as a majority vote amongst the sub-questions’ answers, see Section 4.1.

🔼 This table displays the prompt used in the experiments for the chain-of-thought (CoT) method proposed by Wei et al. (2023). It shows the instructions given to the large language model (LLM) to guide its reasoning process step-by-step, before arriving at a final answer. The prompt is designed to encourage the model to break down the problem into smaller, more manageable parts, thus improving the accuracy and explanation of its reasoning.

read the caption

Table 11: Prompt used for the CoT Wei et al. (2023) experiments.

🔼 This table displays two example question-answer pairs used during the experiments. These examples are employed as few-shot learning examples to guide the model before answering the main questions. The examples cover two different question types to ensure that the model receives broad exposure.

read the caption

Table 12: The two-shot examples used for the CoT Wei et al. (2023) configurations.

🔼 This table details the prompt used for the rephrasing and responding (RaR) experiments in the paper. The RaR technique is a prompting strategy that instructs the language model to first rephrase the given question before attempting to answer it directly. This table outlines the specific instructions given to the model in this experimental setup, thereby providing a clear understanding of how the RaR technique was implemented and utilized in the research.

read the caption

Table 13: Prompt used for the RaR Deng et al. (2024) experiments.

🔼 Table 14 shows the two example questions and answers used in the RaR (Rephrase and Respond) experimental setup. The examples are designed to demonstrate how the RaR prompting technique works by first rephrasing the question to provide more context and detail before answering. This is in contrast to other techniques where the model attempts to answer directly from the initial question. The examples cover two distinct question types, one focused on shared professions and another on identifying a music band, showcasing the versatility of this prompting method across different knowledge domains.

read the caption

Table 14: The two-shot examples used for the RaR Deng et al. (2024) configurations.