Benchmarking Temporal Reasoning and Alignment Across Chinese Dynasties

🔼 Figure 2 presents a statistical overview of the Chinese Time Reasoning (CTM) benchmark dataset. It visually summarizes the distribution of various question types within the dataset and also shows the breakdown of the Timeline Ito Game dataset into different difficulty levels based on the number of entities involved.

read the caption

Figure 2: Statistic of CTM.

🔼 This figure displays the average performance across different LLMs in the Timeline Ito Game. It shows the average accuracy of various LLMs in correctly ordering historical entities based on contextual clues and thematic metaphors. The game involves multiple steps: describing an entity, inferring its rank based on the theme, and determining its order in the timeline. The figure provides a summary of the overall performance, while the detailed results for each LLM are available in Appendix I.

read the caption

Figure 3: Average performance of Ito’s Guessing Game. Detailed results can be found in Appendix I.

🔼 This figure displays the accuracy of various large language models (LLMs) in temporal reasoning tasks, specifically focusing on the ability to correctly identify temporal relationships between entities spanning different Chinese dynasties. The x-axis represents the minimum temporal span (in dynasties) between the entities involved in the task, while the y-axis shows the accuracy of the LLMs’ predictions. The graph illustrates how the difficulty of the task, and thus the model accuracy, changes as the time interval between entities increases. More detailed breakdowns of this accuracy are presented in Figures 23, 24, and 25.

read the caption

Figure 4: Accuracy across entity inter-dynastic intervals under direct prompting setting. The detailed results are shown in Figure 23, Figure 24 and Figure 25.

🔼 This figure compares the performance of different LLMs in temporal reasoning tasks under two settings: close-book (no external knowledge) and open-book (allowing access to external information via search engines). The x-axis represents the different LLMs evaluated. The y-axis displays the accuracy achieved. The bars illustrate the performance difference between the two settings for each LLM, highlighting the impact of access to external information on temporal reasoning capabilities. More detailed results are available in Appendix J.

read the caption

Figure 5: Performance in the close-book and open-book settings. Detailed results can be found in App. J.

🔼 This JSON data represents a structured format for storing information about a historical figure. It includes fields for the person’s name (‘屈原’), the dynasty they lived in (‘先秦’), their place of birth, birth and death years, and a list of their writings and associated sentences from their works. Each work is listed with a title and a specific excerpt. This structured format is used to organize the data in a consistent manner for processing by a computer, allowing for easy data retrieval and analysis.

read the caption

Figure 6: A JSON-format case for historical figure entity.

🔼 The figure displays a JSON-formatted example representing a place entity within the CTM benchmark dataset. The JSON structure includes fields such as ‘dynasty’ (specifying the Chinese dynasty), ‘id’, ‘begin’ and ’end’ (for temporal range), ‘pre_address’ (the modern equivalent of the location), and ‘subordinate_units’ (containing information about administrative divisions within the place during that period). This is a structured way of representing the data for historical places within the dataset, facilitating efficient processing and analysis by machine learning models during evaluation.

read the caption

Figure 7: A JSON-format case for place entity.

🔼 Figure 8 displays a JSON formatted example representing an event entity from the CTM dataset. It shows the structure used to represent events in a standardized way, including fields like ‘id’, ‘dynasty’ (the historical dynasty in which the event occurred), and ‘main_figures’ (key individuals involved). This structured format facilitates data processing and analysis for tasks involving temporal reasoning.

read the caption

Figure 8: A JSON-format case for event entity.

🔼 This table presents the main results of the Question Answering tasks within the Chinese Time Reasoning (CTM) benchmark. It shows the accuracy of various Large Language Models (LLMs) across eight different question types, categorized by the number of temporal entities involved. The best performing model for each task is bolded, while the second-best is underlined, allowing for easy comparison of LLM performance on this challenging temporal reasoning benchmark.

read the caption

Table 2: Main results on QA tasks within CTM benchmark. The best results among all backbones are bolded, and the second-best results are underlined.

🔼 Table 3 presents a detailed statistical overview of the CTM (Chinese Time Reasoning) benchmark dataset. It breaks down the number of samples for each of the eight question-answering tasks (EDD, PJ, TOU, RR, SEC, EEU, TIC, TES, and LSEC) within CTM. Additionally, it provides the distribution of questions based on the number of temporal entities involved. For the Timeline Ito Game component of the benchmark, the table shows the sample counts for the easy, medium, and hard difficulty levels, as categorized by the number of agents involved and cross-temporal entity counts.

read the caption

Table 3: The statistics of CTM.

🔼 This table lists the twelve large language models (LLMs) used in the experiments presented in the paper. It indicates whether each model is open-source or closed-source and provides the model size in billions of parameters.

read the caption

Table 4: LLMs evaluated in our experiments

🔼 This table presents the performance of various Large Language Models (LLMs) on the Timeline Ito Game, a task within the Chinese Time Reasoning (CTM) benchmark. It shows the accuracy (Pass@3 and Pass@8) of each LLM across different difficulty levels of the game, categorized by the number of entities involved (easy, medium, hard). Pass@K indicates whether the correct temporal ordering of entities was achieved within K attempts.

read the caption

Table 5: Main results on Timeline Ito Game within CTM benchmark.

🔼 This table presents a detailed breakdown of the performance of various Large Language Models (LLMs) on the Chinese Time Reasoning (CTM) benchmark, specifically focusing on the results obtained under the ‘open-book’ setting. The open-book setting allows the LLMs to access external knowledge resources during the evaluation, potentially improving their performance on temporal reasoning tasks. The table shows the accuracy of each model across different question types within the CTM benchmark. Each question type assesses different aspects of temporal understanding. The ‘Cross Temp Count’ column likely refers to the number of temporal entities involved in the question, and an ‘Avg.’ column probably reflects an average accuracy across all question types for each model.

read the caption

Table 6: Detailed results under the open-book setting.