Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models

🔼 This figure displays a sample sentence from the OTA-BOUN historical Turkish treebank in the CoNLL-U format. CoNLL-U (CoNLL Universal) is a standard format for representing linguistic annotations, particularly dependency parsing. The table shows each word (token) of the sentence, its part-of-speech tag (POS), its morphological features, and its dependency relation to other words in the sentence. The first column is the index of the word in the sentence. The second column provides the word itself written in the Latin alphabet (a transliteration of the original historical Turkish). The third column shows its part-of-speech tag and morphological features. The last column presents the word in the original Perso-Arabic script used in historical Turkish. The dependency relations are shown via the ‘HEAD’ column, where the index of the head word each word depends on is specified. This detailed annotation makes this figure a useful example of the rigorous annotation process involved in creating the treebank, illustrating the complexity and detail captured in the historical Turkish dataset.

read the caption

Figure 2: CoNLL-U Representation of an example sentence from our OTA-BOUN historical Turkish treebank.

🔼 This figure uses t-SNE (t-distributed Stochastic Neighbor Embedding) to visualize the relationships between documents in the Ottoman Text Corpus. Each point represents a single document, and the color of each point indicates its topic. The clustering of points reveals thematic groups within the corpus. The distribution of these clusters across time (as shown in the accompanying bar charts) demonstrates how topic prevalence varies across different historical periods. This visualization therefore effectively displays both the topical and temporal diversity of the Ottoman Text Corpus.

read the caption

Figure 3: t-SNE visualization of documents in the Ottoman Text Corpus: Each point represents a document, color-coded by topic. This visualization highlights thematic clusters within the corpus and shows how topics are distributed periodically, complementing the map’s representation of vocabulary diversity and topic.

🔼 Table 2 presents a detailed statistical overview of the Ottoman Turkish-Boğaziçi University (OTA-BOUN) historical treebank, a valuable linguistic resource for researchers working with historical Turkish. The table summarizes key features of the treebank, providing insights into the nature and characteristics of the data. Specifically, it provides the total number of sentences, tokens, and the average token count per sentence, giving a sense of the corpus size and sentence length distribution. It also presents the number of unique parts-of-speech (POS) tags and unique morphological features, offering insights into the richness and complexity of the language represented in the treebank. Finally, it includes the number of unique dependency relations, which are crucial for understanding the grammatical structures captured within the treebank.

read the caption

Table 2: Some statistics of the OTA-BOUN historical Turkish treebank

🔼 This table presents a detailed breakdown of the dependency relations found within the Ottoman Turkish Universal Dependencies (OTA-BOUN) treebank. It lists each dependency relation type, its frequency count, and its percentage relative to the total number of dependency relations in the treebank. This provides valuable insights into the syntactic structures characteristic of historical Turkish, such as the prevalence of particular types of relationships between words in sentences.

read the caption

Table 3: Counts and percentages of dependency relation types in the OTA-BOUN treebank

🔼 Table 4 presents a quantitative comparison of the OTA-BOUN historical Turkish treebank with two widely used modern Turkish treebanks (TR-BOUN and IMST-UD). The comparison focuses on key metrics related to token and dependency features, offering insights into the structural differences between historical and modern Turkish. Metrics include the average number of tokens per sentence, the percentage of conjunct (conj) dependency relations, and the percentage of light verb compound (compound:lvc) dependency relations and adnominal clause (acl) relations. These metrics provide valuable insights into the syntactic and stylistic characteristics of each treebank, highlighting how historical Turkish differs from modern Turkish in terms of sentence structure and grammatical constructions.

read the caption

Table 4: Comparison of historical Turkish treebank with the two most frequently used modern Turkish treebanks in terms of token and dependency metrics

🔼 This table presents examples of text extraction errors encountered during the digital conversion of historical Turkish documents. It showcases the types of errors, their causes (e.g., diacritical encoding, script conversion, word segmentation, character substitution, mixed script issues), and analysis of why these errors occurred during the digitization process. Each row details a specific instance with the original (expected) text, the extracted text containing errors, and a breakdown explaining the nature and source of the error, including references to Unicode issues and challenges related to processing historical Turkish script variations.

read the caption

Table 5: Analysis of sample text extraction errors in digital conversion of historical Turkish documents

🔼 Table 6 presents the performance of three different pre-trained Named Entity Recognition (NER) models (BERTurk, mBERT, and TURNA) on the HisTR dataset. The HisTR dataset is specifically designed for historical Turkish texts, posing unique challenges not encountered in modern Turkish NER. The table shows the performance of the models on both an in-domain development set and an out-of-domain test set. The in-domain data consists of similar texts to those in the training data, while the out-of-domain data has different characteristics. Multiple experiments were conducted for each model using various combinations of training data. The results are reported using the precision, recall, and F1-score for each setting, offering insights into how well each model generalizes to the different datasets.

read the caption

Table 6: The overall performance of BERTurk, mBERT, and TURNA NER models on the in-domain development and out-of-domain test sets of the HisTR dataset when using different combinations of fine-tuning sets

🔼 Table 7 presents the results of dependency parsing and POS tagging experiments using the STEPS parser with BERTurk and mBERT models. It shows the performance of different model configurations (fine-tuned on various combinations of the OTA-BOUN and TR-BOUN treebanks) on both historical (OTA-BOUN) and modern (TR-BOUN) Turkish datasets. The table includes metrics such as unlabeled attachment score (UAS), labeled attachment score (LAS), and Universal Part-of-Speech (UPOS) F1-score, providing a comprehensive evaluation of the models’ ability to accurately identify dependency relations and POS tags in both historical and modern Turkish text.

read the caption

Table 7: The overall performance of BERTurk- and mBERT-based models on the test sets of the OTA-BOUN and TR-BOUN treebanks, presented for different combinations of fine-tuning sets. UAS and LAS represent unlabeled and labeled attachment scores, respectively, which are used to evaluate the models’ performance in constructing dependency relations. UPOS F1 refers to the F1 score of the sequence tagger models in predicting the universal POS tags of words in the corresponding test sets.