RuCCoD: Towards Automated ICD Coding in Russian

🔼 This figure illustrates the two main tasks addressed in the paper: ICD coding and diagnosis prediction. The ICD coding task (shown in blue) involves using a doctor’s diagnostic conclusion (from the current visit) to assign ICD codes. An AI model is trained to perform this task, generating AI-assigned ICD codes. The diagnosis prediction task (shown in yellow) predicts likely ICD codes based on a patient’s complete medical history, excluding the doctor’s conclusion from the current visit. Both the original ICD codes (assigned by doctors) and the AI-generated ICD codes are used as training targets for the diagnosis prediction models, enabling comparison and improvement of AI performance.

read the caption

Figure 2: Schematic description of ICD coding (in blue) and diagnosis prediction tasks (in yellow). Diagnosis prediction uses prior EHR data and current visit details, excluding the doctor’s conclusion, which is used for ICD coding to generate AI-assigned ICD codes. Both original and AI ICD code lists are then used as targets to train different diagnosis prediction models.

🔼 This histogram displays the frequency distribution of ICD codes within the RuCCoD training dataset. The x-axis represents ICD codes sorted by their frequency of appearance in the dataset, and the y-axis shows the number of times each code appears. The graph reveals a highly skewed distribution, with a relatively small number of codes appearing very frequently, and a large number of codes appearing very infrequently, reflecting the uneven distribution of diagnoses in real-world clinical data.

read the caption

Figure 3: Distribution of ICD code frequencies in the RuCCoD train set.

🔼 This figure shows the performance comparison between two models trained for diagnosis prediction on a manual test set. One model was trained using the original dataset (manually annotated), while the other was trained using a linked dataset (automatically annotated using ICD codes generated by a model). The x-axis represents the training steps, and the y-axis represents the weighted F1-score, a metric that accounts for class imbalances in the dataset. The graph illustrates how the weighted F1-score changes as the models are trained over different numbers of steps. It shows that the model trained on the automatically annotated dataset performs significantly better than the model trained on the original dataset.

read the caption

Figure 4: Comparison of weighted F1 scores on the manual diagnosis prediction test set for models trained on original and linked datasets at different training steps.

🔼 This figure displays the F1-score distribution for the top and bottom 10% most frequent ICD codes within a common test set, comparing performance of the model trained on the original dataset versus the linked dataset. The top 10% represents the most frequently occurring ICD codes, while the bottom 10% represents the least frequent codes, with a minimum frequency threshold of 15 instances within the test set. This visualization highlights the effect of training data (original vs. linked) on the model’s ability to predict both frequent and infrequent disease codes.

read the caption

Figure 5: F1 score distribution for top and bottom 10% frequent ICD codes in the common test set.

🔼 This table presents a statistical overview of the RuCCoD-DP dataset, which is used for diagnosis prediction. RuCCoD-DP is a collection of real-world electronic health records (EHRs) divided into training and testing sets. The table shows the number of records, unique patients, and unique ICD codes in each set. It also provides the average number of ICD codes per patient, the average number of EHR records per patient before the current appointment, the average length of EHR records per appointment, the average age of patients, and the percentage of male patients. The values in parentheses represent the 25th, 50th, and 75th percentiles, giving a clearer picture of the data distribution.

read the caption

Table 2: Statistics for the randomly split training and testing sets of RuCCoD-DP for diagnosis prediction. Values in brackets show the 25th, 50th, and 75th percentiles.

🔼 This table presents the performance of different models on the entity-level code assignment task using the RuCCoD test set. The models evaluated include a BERT-based pipeline, LLMs with PEFT, and LLMs with RAG. The metrics reported are precision, recall, F1-score, and accuracy, allowing for a comprehensive comparison of model effectiveness. The ‘best’ performing model for each metric is highlighted in bold. Further experimental results using different LLMs, corpora, and terminologies are detailed in the Appendix (sections D, E, and F).

read the caption

Table 3: Entity-level code assignment metrics on RuCCoD’s test set. The best results are highlighted in bold. We also refer to Appx. D, E, F on more experiments with different LMs, corpora, and terminologies.

🔼 This table details the models and training hyperparameters used in the experiments. It lists the learning rate (LR), number of epochs, batch size (BS), scheduler type used for adjusting learning rate during training, and weight decay (WD) values for each model. The models are categorized by task (NER, EL, and ICD code prediction). Understanding these hyperparameters is crucial for replicating and interpreting the experimental results.

read the caption

Table 4: Models and training hyperparameters. LR stands for learning rate, BS for batch size, WD for weight decay

🔼 This table presents the performance of different models on the Named Entity Recognition (NER) task using the RuCCoD dataset. It shows the F1-score, precision, and recall achieved by various models trained on different combinations of training data. The models evaluated include RuBioBERT (with and without BINDER), highlighting their effectiveness in extracting relevant entities from clinical texts in the Russian language.

read the caption

Table 5: Evaluation results for NER task on RuCCoD dataset.

🔼 This table presents the results of experiments evaluating the effectiveness of transfer learning in biomedical entity linking. Four different models (SapBERT, CODER, BERGAMOT) were trained using various combinations of datasets: RuCCoD (Russian ICD Coding Dataset), RuCCON, and NEREL-BIO. Each model was tested with two evaluation methods: ‘strict’ (exact match between predicted and ground truth codes) and ‘relaxed’ (truncated codes to higher level). The results show the precision, recall, F1-score, and accuracy for each model and dataset combination under both strict and relaxed evaluation schemes. One data setting, ICD+UMLS synonyms, involves enriching the training data with disease name synonyms from the UMLS knowledge base to assess the impact of vocabulary expansion.

read the caption

Table 6: Cross-domain transfer results for biomedical linking models. Evaluation results for linking models trained on RuCOD, RuCCoN, NEREL-BIO as well as their union. ICD+UMLS synonyms stands for ICD train set with the vocabulary enriched with ICD disease name synonyms from the UMLS knowledge base. The best results for each model and set-up are highlighted in bold.

🔼 This table presents the performance of several fine-tuned large language models (LLMs) on the RuCCoD dataset for the task of ICD (International Classification of Diseases) coding. The models were evaluated using micro-averaged precision, recall, F1-score, and accuracy. The table shows the results broken down by the model used and the different corpora employed during training (ICD dict., ICD dict.+RuCCoD, ICD dict.+BERGAMOT). The best-performing model and corpus combination for each metric are highlighted in bold, allowing for a direct comparison across various LLMs and training data configurations.

read the caption

Table 7: ICD coding results for finetuned LLMs on RuCCoD. The best results are highlighted in bold.

🔼 This table presents the performance of a BERT-based information extraction (IE) pipeline on the RuCCoD corpus for three entity-level tasks: Named Entity Recognition (NER), NER + Entity Linking, and ICD Code Assignment. The pipeline uses various combinations of pre-trained models and training corpora (RuCCoD, NEREL-BIO, RuCCON, and BioSyn) for NER and entity linking. The results are shown as precision, recall, F1-score, and accuracy metrics, highlighting the best-performing configurations for each task. This table demonstrates the impact of different model choices and training data on the accuracy of extracting and linking disease-related entities to ICD codes.

read the caption

Table 8: Evaluation results for entity-level tasks for BERT-based IE pipeline on RuCCoD corpus. The best results are highlighted in bold.

🔼 This table presents the performance of different Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) on three tasks related to ICD coding: Named Entity Recognition (NER), linking entities to ICD codes, and end-to-end entity linking. It shows precision, recall, F1-score, and accuracy for each LLM on each task, using different knowledge sources (ICD dictionary, RuCCOD dataset) for the RAG component.

read the caption

Table 9: Evaluation results for NER, Code assignment, and end-to-end entity linking task on RuCCoD for LLM+RAG pipeline.

🔼 This table presents the performance of different Large Language Models (LLMs) with Retrieval Augmented Generation (RAG) on three tasks related to ICD coding: Named Entity Recognition (NER), NER+linking, and code assignment. The evaluation was performed on the RuCCoD dataset, using either NEREL-BIO or RuCCoN as vectorstores. The results show precision, recall, F-score, and accuracy for each model and task. This helps assess the efficacy of different LLMs and approaches in automating various stages of the ICD coding process.

read the caption

Table 10: Evaluation results for NER, Code assignment, and end-to-end entity linking task on RuCCoD for LLM+RAG pipeline using NEREL-BIO and RuCCoN for vectorstore.

🔼 This table presents the results of experiments evaluating the performance of different Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) on the RuCCoD dataset. The evaluation used a relaxed scoring approach, focusing on the NER (Named Entity Recognition), code assignment, and end-to-end entity linking tasks. The models are compared based on their precision, recall, F1-score, and accuracy, with results shown for various configurations, including the use of different dictionaries and datasets in the RAG pipeline. The results are analyzed to understand the effectiveness of each model in the relaxed setting.

read the caption

Table 11: Relaxed evaluation results for NER, Code assignment, and end-to-end entity linking task on RuCCoD for LLM+RAG pipeline.

🔼 Table 12 presents the relaxed evaluation metrics for three tasks: Named Entity Recognition (NER), ICD code assignment, and end-to-end entity linking. The evaluation is performed on the RuCCoD dataset using the LLM+RAG pipeline. Specifically, the results showcase the performance of several large language models (LLMs) in these tasks when using either the NEREL-BIO or RuCCoN dataset as a vector store for retrieval augmented generation (RAG). The metrics presented include precision, recall, F-score, and accuracy, offering a comprehensive view of the models’ performance under relaxed evaluation conditions.

read the caption

Table 12: Relaxed evaluation results for NER, Code assignment, and end-to-end entity linking task on RuCCoD for LLM+RAG pipeline using NEREL-BIO and RuCCoN for vectorstore.