Typhoon T1: An Open Thai Reasoning Model

🔼 This figure illustrates three different formats for representing the reasoning process of a large language model (LLM). Panel (a) shows unstructured thinking, where the LLM’s reasoning and final answer are presented as a single, continuous text stream without any explicit separation or structure. Panel (b) demonstrates semi-structured thinking, which adds simple XML tags such as and to delineate the reasoning steps from the final answer. Panel (c) shows structured thinking, a more advanced format that utilizes additional XML tags within the section to further organize and structure the LLM’s reasoning process, helping to clarify the intermediate steps and sub-goals involved in arriving at the final answer. This structured format is designed to improve the clarity and organization of the LLM’s thinking.

read the caption

Figure 2: Differences between three thinking formats: (a) Unstructured thinking, where no XML structural tags are included; (b) Semi-structured thinking, which is similar to unstructured thinking but adds and tags to separate thoughts and responses; (c) Structured thinking, which introduces additional XML tags for structural purposes in the thoughts section.

🔼 This figure presents the results of an experiment assessing the impact of training data size on the performance of a reasoning model. Multiple datasets were used, and varying percentages of the training data (5%, 10%, 25%, 50%, 75%, and 100%) were used to train separate models. The results show that for most datasets, using more than 75% of the training data did not significantly improve performance, and in some cases even led to performance degradation. The exception is the GSM8K dataset, which demonstrated a consistent performance improvement as more training data was used.

read the caption

Figure 3: Increasing the proportion of the training set beyond 75% results in performance degradation for some datasets, while GSM8K generally shows a trend of performance improvement with the proportion.

🔼 Figure 4 presents a comparative analysis of the performance of three different models: Typhoon T1-EN, Typhoon T1, and the baseline Typhoon 2 3B Instruct model, across six distinct evaluation benchmarks. It visually represents the relative performance of each model on each benchmark, enabling easy comparison of their strengths and weaknesses. The benchmarks likely represent a variety of tasks to assess the models’ reasoning capabilities comprehensively.

read the caption

Figure 4: Final performance comparison of Typhoon T1-EN and Typhoon T1 against the baseline Typhoon T1 3B Instruct model across six evaluation benchmarks.

🔼 This pie chart visualizes the distribution of the training data across five different domains used in the experiments described in the paper. Each slice of the pie represents a domain, and its size corresponds to the proportion of data samples from that domain within the overall training dataset. The domains are: Mathematics, Instruction Following, Coding, Safety, and Finance.

read the caption

Figure 5: This figures show domain distribution of the training set for the experiments.

🔼 This figure presents a side-by-side comparison of the reasoning traces generated by two models: Typhoon T1 and Typhoon T1-EN. Typhoon T1, trained with Thai-translated data, produces a reasoning trace in Thai, showcasing its ability to generate reasoning steps in a low-resource language. In contrast, Typhoon T1-EN, trained without Thai data, provides a reasoning trace in English. The comparison highlights the impact of training data on the model’s ability to perform reasoning and generate traces in different languages. Both models answer the same question, allowing for a direct comparison of their respective reasoning processes and language usage.

read the caption

Figure 6: This figure shows Typhoon T1’s Thai thinking trace and Typhoon T1-EN’s English thinking trace.

🔼 This table compares the performance of different language models across six benchmarks: GSM8K, HumanEval+, IFEval, GPQA, MMLU Pro, and ThaiExam. The models compared include the baseline Typhoon 2 3B Instruct (Typhoon 2), variants trained with different thinking methods (zero-shot, zero-shot CoT, and supervised fine-tuning, SFT), and the Typhoon T model trained on long-thinking datasets. Higher scores indicate better performance. Bold values highlight the best score for each benchmark. Underlined scores show improvements compared to the Typhoon 2 3B Instruct baseline. The results demonstrate that all reasoning models perform better than SFT on the original dataset.

read the caption

Table 2: Performance of models on each benchmark (higher is better). Bold indicates the best score in each column. Underlined scores denote improvements over the baseline, Typhoon 2 3B Instruct. We apply this convention across all results tables. Typhoon 2 refers to Typhoon 2 3B Instruct, and Typhoon T refers to its variant trained on a long-thinking dataset. SFT refers to supervised fine-tuning on the original datasets. All reasoning models show improvement over SFT on the original dataset.

🔼 This table presents the performance of different model variants on various benchmark datasets. The models were trained with varying amounts of additional data, including a baseline model (Typhoon T1-EN) and variants trained with an additional 1.5k samples and with continual supervised fine-tuning (CSFT). The results show the impact of the additional training data on the model’s performance across several reasoning tasks, revealing that adding 1.5k samples improved scores on instruction following (IFEval) and the Thai language exam (ThaiExam). Continual SFT, however, led to a significant reduction in overall performance.

read the caption

Table 3: Performance of model variants on various benchmarks, evaluating the impact of additional training data. CSFT refers to continual SFT. Adding 1.5k samples improves IFEval and ThaiExam scores, while CSFT significantly reduces overall performance.

🔼 This table presents the results of an experiment where the multilingual reasoning model Typhoon T1 was constrained to reason in either English or Thai, and its performance was compared against when it was allowed to choose its own reasoning language. The results show that constraining the model to a specific language (either English or Thai) leads to a decrease in overall accuracy across various benchmarks. Although English-forced reasoning performed slightly better than Thai-forced reasoning, allowing the model to select its own language resulted in the highest accuracy, demonstrating the importance of flexibility in multilingual reasoning.

read the caption

Table 4: EN denotes forced reasoning in English, and TH denotes forced reasoning in Thai. Constraining Typhoon T1 to reason in a specific language degrades overall accuracy. English reasoning is slightly more effective than Thai reasoning across most benchmarks. However, allowing the model to choose its own thinking language yields the best performance.

🔼 Table 5 details the composition of the training dataset used to develop the Typhoon T1 reasoning model. It breaks down the number of records from each dataset used across five different domains: mathematics, instruction following, coding, safety, and finance. The table provides context on the diversity of tasks and data sources used for model training, highlighting the quantity of data drawn from each source dataset within each domain.

read the caption

Table 5: Data mixture of the training set.

🔼 This table presents the average number of tokens generated by different language models across various benchmark datasets. The models include Typhoon 2 (with zero-shot and zero-shot chain-of-thought prompting) and Typhoon T (with unstructured, semi-structured, and structured thinking formats). The benchmarks show the average token count for each model configuration, providing insights into the length and complexity of the generated responses.

read the caption

Table 6: Average number of output tokens generated by each model on the benchmarks.

🔼 This table presents the performance of reasoning models trained on different sizes of the training dataset, ranging from 5% to 100%. The results are evaluated across six benchmarks: GSM8K, HumanEval+, IFEval, GPQA, MMLU Pro, and ThaiExam. The table shows that smaller datasets sometimes achieve better performance than the full dataset, especially on the GPQA and MMLU Pro benchmarks. This suggests that there is a potential optimal dataset size for training these models, and that overtraining on very large datasets might not necessarily result in improved performance.

read the caption

Table 7: Performance at different dataset sizes. Smaller dataset sizes can sometimes outperform the 100% baseline, particularly in GPQA and MMLU Pro.

🔼 This table presents the results of a leave-one-out experiment designed to analyze the impact of removing specific domains from the training dataset on the model’s performance across various benchmarks. Each row represents a model trained without a particular domain (indicated by ‘-’). The columns show the performance metrics (GSM8K, HumanEval+, IFEval, GPQA, MMLU Pro, ThaiExam) for each model variant. Red values highlight the largest performance drop observed after excluding a specific domain. The results demonstrate that the removal of certain domains, like mathematical reasoning, significantly affects some benchmarks (IFEval) while surprisingly increasing performance on others (MMLU Pro after removing the safety domain).

read the caption

Table 8: Leave-one-out experiment results, assessing the impact of removing specific domains from training. red values highlight the largest performance drop in each column. The “-” symbol denotes the removal of the corresponding domain from training. Excluding mathematical reasoning strongly improves IFEval performance, while safety removal boosts MMLU Pro.