NeoBERT: A Next-Generation BERT

🔼 Figure 2 presents a comparison of the pseudo-perplexity scores achieved by two versions of the NeoBERT model – NeoBERT1024 and NeoBERT4096 – across varying sequence lengths. Pseudo-perplexity serves as a measure of how well the model predicts the next token in a sequence; lower scores indicate better performance. The left panel shows NeoBERT1024’s performance, trained with a maximum sequence length of 1024 tokens. The right panel shows NeoBERT4096, which underwent an additional training phase with longer sequences (up to 4096 tokens). The figure demonstrates that extending the pre-training with longer sequences significantly improves the NeoBERT model’s ability to handle and generate longer sequences accurately, as evidenced by the lower perplexity scores for NeoBERT4096, particularly at longer sequence lengths.

read the caption

Figure 2: Pseudo-Perplexity in function of the sequence length for NeoBERT1024 (left) and NeoBERT4096 (right). This validates the effectiveness of the final pre-training stage on NeoBERT’s ability to model long sequences.

🔼 Figure 3 illustrates the throughput (tokens processed per second) of various language models as the sequence length increases. The models compared are BERTbase, ROBERTabase, BERTlarge, ROBERTalarge, NeoBERT, ModernBERTbase, and ModernBERTlarge. The x-axis represents the sequence length, and the y-axis represents the throughput. The figure shows that NeoBERT, despite having 100 million more parameters than ModernBERTbase, achieves a significantly higher throughput when the sequence length exceeds 1024 tokens. This highlights NeoBERT’s efficiency in handling long sequences.

read the caption

Figure 3: Model throughput (tokens per second) as a function of sequence length (↑↑\uparrow↑ is better). Above 1,02410241,0241 , 024 in sequence length, NeoBERT surpasses ModernBERTbase despite having 100⁢M100𝑀100M100 italic_M more parameters.

🔼 This figure displays the performance of BERT and RoBERTa models on the English subset of the MTEB benchmark without any fine-tuning. It demonstrates the zero-shot performance of these models, meaning their performance is evaluated directly after pre-training without any task-specific adaptation. The graph likely shows the average score across multiple tasks within the MTEB benchmark, indicating the models’ inherent abilities to handle various tasks before any further training or optimization.

read the caption

Figure 4: Zero-shot evaluation of BERT and RoBERTa on the English subset of MTEB.

🔼 This table details the modifications made during a series of ablation experiments to improve a BERT-like model, ultimately resulting in NeoBERT. It shows the changes introduced iteratively to the base model (M0, similar to BERT) in each step (M1-M9), highlighting modifications to embeddings, activation functions, normalization, datasets, tokenizers, optimizers, schedulers, masking schemes, model size, and context length. The final model, M9, represents NeoBERT.

read the caption

Table 2: Modifications between successive ablations. The initial M⁢0𝑀0M0italic_M 0 baseline corresponds to a model similar to BERT, while M⁢9𝑀9M9italic_M 9 corresponds to NeoBERT.

🔼 This table presents the GLUE (General Language Understanding Evaluation) benchmark scores achieved by various language models on their development sets. It compares the performance of NeoBERT against several established models including BERT, RoBERTa, DeBERTa, NomicBERT, GTE, and ModernBERT. The scores are broken down by individual tasks within the GLUE benchmark, allowing for a detailed comparison of each model’s strengths and weaknesses across different NLP tasks. The table also indicates the size (in parameters) of each model, showing how NeoBERT’s performance compares even with smaller model size.

read the caption

Table 3: GLUE scores on the development set. Baseline scores were retrieved as follows: BERT from Table 1 of Devlin et al. (2019), RoBERTa from Table 8 of Liu et al. (2019), DeBERTa from Table 3 of He et al. (2023), NomicBERT from Table 2 of Nussbaum et al. (2024), GTE from Table 13 of Zhang et al. (2024), and ModernBERT from Table 5 of Warner et al. (2024).

🔼 This table presents the results of the MTEB (Massive Text Embedding Benchmark) English subset evaluation. Multiple pre-trained language models were fine-tuned using a contrastive learning approach for 2000 steps. The table shows the performance of each model across seven different tasks within the benchmark (Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity (STS), and Summarization), along with the average score across all tasks. The models are categorized by size (Base, Medium, Large), providing a comparison of performance across different model scales.

read the caption

Table 4: MTEB scores on the English subset after 2,000 steps of fine-tuning with contrastive learning.

🔼 Table 5 presents the optimal hyperparameters found through a grid search for fine-tuning the NeoBERT model on the GLUE benchmark. The search explored various combinations of batch sizes (2, 4, 8, 16, 32), learning rates (5e-6, 6e-6, 8e-6, 1e-5, 2e-5, 3e-5), and weight decay values (1e-2, 1e-5) for each of the GLUE tasks. The table lists the optimal settings discovered for each task, aiding reproducibility and comparison of results.

read the caption

Table 5: Optimal hyperparameters for GLUE tasks. The grid search was conducted over batch sizes {2,4,8,16,32}2481632\{2,4,8,16,32\}{ 2 , 4 , 8 , 16 , 32 }, learning rates {5⁢e−6,6⁢e−6,8⁢e−6,1⁢e−5,2⁢e−5,3⁢e−5}5𝑒66𝑒68𝑒61𝑒52𝑒53𝑒5\{5e-6,6e-6,8e-6,1e-5,2e-5,3e-5\}{ 5 italic_e - 6 , 6 italic_e - 6 , 8 italic_e - 6 , 1 italic_e - 5 , 2 italic_e - 5 , 3 italic_e - 5 }, and weight decay values {1⁢e−2,1⁢e−5}1𝑒21𝑒5\{1e-2,1e-5\}{ 1 italic_e - 2 , 1 italic_e - 5 }.

🔼 This table details the instructions used for fine-tuning various pre-trained models on different contrastive learning datasets. Each row represents a dataset, specifying the task and the instructions given to the model for that task. The instructions provide context to the models, guiding them on how to process the data and generate appropriate outputs. The information is crucial for understanding the fine-tuning process and how the models were prepared for the downstream evaluations.

read the caption

Table 6: Instructions for fine-tuning on the different contrastive learning datasets.

🔼 This table details the specific instructions used for evaluating model performance on each of the sub-tasks within the MTEB benchmark. For each task, it provides a description outlining the input format and the expected output, clarifying the nature of the prediction required from the language model.

read the caption

Table 7: Instructions for evaluation on the different MTEB tasks.

🔼 This table lists instructions for evaluating various tasks within the MTEB (Massive Text Embedding Benchmark). Each row represents a different task, specifying the type of input given (e.g., a question, a review, a news summary) and what the model is expected to retrieve or classify in response (e.g., relevant documents, sentiment, intents). The table provides a comprehensive overview of the diverse tasks included in MTEB, showing the range of natural language understanding abilities being assessed by the benchmark.

read the caption

Table 8: Instructions for evaluation on the different MTEB tasks.

🔼 This table presents the throughput, measured in thousands of tokens processed per second, for different language models at various sequence lengths. The throughput is determined using the optimal batch size for each model and sequence length combination. This allows for a comparison of the efficiency of each model in handling different input sizes, which is critical for real-world applications where processing speed is often a major constraint. The models are grouped by size (base, medium, large).

read the caption

Table 9: Throughput (103superscript10310^{3}10 start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT tokens / second) in function of the sequence length, with optimal batch size.