GKG-LLM: A Unified Framework for Generalized Knowledge Graph Construction

🔼 Figure 2 shows the distribution of the datasets used for training and evaluating the Generalized Knowledge Graph (GKG) construction task. It visually represents the various sub-tasks involved in constructing three types of graphs: Knowledge Graphs (KGs), Event Knowledge Graphs (EKGs), and Commonsense Knowledge Graphs (CKGs). Each sub-task is associated with one or more datasets, indicating the data sources utilized for that specific task. This figure provides a comprehensive overview of the data utilized for training and testing in the study, highlighting the diverse range of tasks and datasets involved in the GKG construction process.

read the caption

Figure 2: The illustration of the data distribution for all GKG sub-tasks.

🔼 This figure illustrates the three-stage curriculum learning framework used to train the GKG-LLM model. The top section shows the unified GKG dataset (𝒟G) comprising data from KG, EKG, and CKG sub-tasks, as well as a counter task dataset. The bottom section details the three training stages: 1) KG Empowerment: foundational skills are built using KG datasets; 2) EKG Enhancement: specific capabilities are enhanced using EKG datasets; and 3) CKG Generalization: generalization is achieved using CKG and counter task datasets. Thick arrows indicate the transfer of model parameters between stages, starting from a base model to create the G-Micro, G-Mid, and finally, the GKG-LLM model.

read the caption

Figure 3: Three-stage curriculum learning tuning framework of GKG-LLM. The upper part represents the GKG dataset 𝒟Gsubscript𝒟𝐺\mathcal{D}_{G}caligraphic_D start_POSTSUBSCRIPT italic_G end_POSTSUBSCRIPT, consisting of the unified datasets. The lower part shows the three stages of GKG training: the KG empowerment stage using the KG datasets to build foundational skills, the EKG enhancement stage using the EKG datasets to enhance specific capabilities, and the CKG generalization stage using the CKG datasets and the counter task dataset to achieve generalization of the GKG-LLM capabilities. The thick arrows between the stages represent the delivery of model parameters from base model to each version of GKG-LLM.

🔼 This figure displays the average performance across all tasks for three different fine-tuning orders: K-E-C (KG, EKG, then CKG), K-C-E (KG, CKG, then EKG), E-K-C (EKG, KG, then CKG), E-C-K (EKG, CKG, then KG), C-K-E (CKG, KG, then EKG), and C-E-K (CKG, EKG, then KG). The results show varying performance across the different sequences, demonstrating the impact of the chosen order on overall model performance. The K-E-C sequence used in the main study demonstrates the best overall performance, supporting the study’s methodology and indicating that a progressive training approach, starting with simpler knowledge graphs and progressing to more complex ones, yields the best results.

read the caption

Figure 4: Results of different fine-tuning orders. “K-E-C” means the fine-tuning order is KG, EKG and CKG. The following sets of experiments are similar to this one.

🔼 Figure 5 displays the performance comparison between using a single type of graph for fine-tuning and the three-stage fine-tuning approach of GKG-LLM. The graph shows performance across KG, EKG, and CKG sub-tasks. It demonstrates that the three-stage curriculum learning process (KG empowerment, EKG enhancement, CKG generalization) progressively improves the model’s capability across all three graph types, significantly outperforming the single-graph fine-tuning method.

read the caption

Figure 5: Fine-tuning with a single type of graph and performance of different intermediate version in the GKG-LLM.

🔼 This figure displays the impact of varying training dataset sizes on the model’s performance. The x-axis represents the percentage of the complete dataset used for training (10%, 20%, 40%, 60%, 80%, and 100%). The y-axis shows the average F1 score achieved across all tasks. The figure demonstrates the relationship between the amount of training data and the model’s performance across three types of knowledge graph sub-tasks (KG, EKG, CKG), and an overall GKG average. It helps illustrate the point of diminishing returns in terms of performance gain as training data increases.

read the caption

Figure 6: Results of training with different proportions of complete data.

🔼 Figure 7 presents the average performance of various models on out-of-distribution (OOD) datasets. The OOD datasets used are TCR, Causal-TB, and R8, each representing a distinct and challenging test scenario. The figure displays the average F1 score (or other relevant metric) achieved by each model on these three datasets, allowing for a direct comparison of their robustness and generalization capabilities in handling unseen data.

read the caption

Figure 7: The average performance on OOD datasets, consisting TCR, Causal-TB and R8 datasets.

🔼 Figure 8 shows an example data entry from the WIKIEVENTS dataset, illustrating the standardized format used throughout the paper. Each data entry consists of five key fields: ‘ID’ uniquely identifies the specific data point; ‘instruction’ provides the task instructions given to the model; ‘few-shot/zero-shot’ indicates whether few-shot learning examples were provided; ‘input’ contains the input data given to the model for processing; and ‘output’ shows the expected output or ground truth. This standardized format enables the unified processing of diverse sub-tasks within the generalized knowledge graph (GKG) construction framework.

read the caption

Figure 8: An example from the WIKEVENTS dataset. It consists of five fields I⁢D𝐼𝐷IDitalic_I italic_D, instruction sisubscript𝑠𝑖s_{i}italic_s start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, few-shot f⁢s𝑓𝑠fsitalic_f italic_s / zero-shot z⁢s𝑧𝑠zsitalic_z italic_s , input xisubscript𝑥𝑖x_{i}italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and output yisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT.

🔼 Figure 9 displays a bar chart comparing the performance of three different model training strategies against the performance of the GKG-LLM model. The three strategies represent progressively more complex training approaches. The chart shows the average F1 scores achieved on three different types of knowledge graph sub-tasks (KG, EKG, CKG) for each model. This helps visualize how a three-stage curriculum learning approach enhances the model’s ability to handle various tasks compared to single-stage training methods.

read the caption

Figure 9: Comparison of Results by different settings and GKG-LLM.

🔼 This heatmap visualizes the performance of the GKG-LLM model under different hyperparameter settings for the LoRA+ fine-tuning technique. The x-axis represents the value of ηB (eta_B), and the y-axis represents the value of ηA (eta_A). Each cell in the heatmap shows the model’s performance (likely F1-score or accuracy) on the GKG construction task under the corresponding ηA and ηB values. The color intensity represents the performance level, with darker colors indicating better performance. This figure is crucial for determining the optimal hyperparameter combination to maximize the model’s effectiveness.

read the caption

Figure 10: Heatmap of Scores for Different ηAsubscript𝜂𝐴\eta_{A}italic_η start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT and ηBsubscript𝜂𝐵\eta_{B}italic_η start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT Values for our training strategy.

🔼 This table compares the performance of four different prompting strategies on various tasks. The strategies are: using full prompts (𝒫), a single instruction regardless of diversity (𝒫si), zero-shot prompting only (𝒫zs), and combining a single instruction with zero-shot prompting (𝒫si+zs). The results are presented in terms of evaluation metrics (specified elsewhere in the paper) for each prompt strategy, showing how different instruction methods impact model performance.

read the caption

Table 2: Performance comparison of different prompt strategies on the evaluation metrics. 𝒫𝒫\mathcal{P}caligraphic_P denotes full prompts, 𝒫sisubscript𝒫si\mathcal{P}_{\text{si}}caligraphic_P start_POSTSUBSCRIPT si end_POSTSUBSCRIPT refers to a single instruction regardless of diversity, 𝒫zssubscript𝒫zs\mathcal{P}_{\text{zs}}caligraphic_P start_POSTSUBSCRIPT zs end_POSTSUBSCRIPT represents zero-shot only, and 𝒫si+zssubscript𝒫si+zs\mathcal{P}_{\text{si+zs}}caligraphic_P start_POSTSUBSCRIPT si+zs end_POSTSUBSCRIPT combines single instruction with zero-shot prompting.

🔼 This table details the datasets used in the paper, categorized by graph type (KG, EKG, CKG) and sub-task. It shows the number of training and testing samples for each dataset, whether the dataset was sampled to balance the data, whether it was held out (not used during training), and the original source of the dataset.

read the caption

Table 3: Detailed illustrations of 15 sub-task types across 29 datasets, categorized within three types of graphs, along with a counter dataset—WebNLG. # Train and # Test represent the number of training and testing samples, respectively. Sampled? indicates whether the dataset is sampled from the original to achieve data balancing. Held-out? specifies whether the dataset is used during the training phase. Original Source refers to the citation of the original paper.