Large Language Models

New data-free methodology creates effective, generalizable LLMs guardrails against off-topic prompts, significantly improving LLM safety and responsible use.

UltraMem, a novel ultra-sparse memory network, drastically speeds up LLM inference by 6x compared to MoE while maintaining performance, paving the way for efficient large-scale model deployment.

RedPajama, two massive open-source datasets, are released for training LLMs, improving transparency and facilitating the development of high-performing open-source models.

SUTRA tokenizer outperforms other LLMs in Indian languages, improving efficiency and facilitating better model performance.

Verifier engineering: A new post-training paradigm for foundation models using automated verifiers to provide effective supervision signals, enhancing capabilities beyond traditional data-centric meth…

SageAttention2 achieves 4-bit accurate attention, boosting inference speed by 2x compared to FlashAttention2, while maintaining end-to-end accuracy across diverse models.

New German-only LLMs, LLäMmlein 120M & 1B, trained from scratch & openly released, show competitive performance and offer insights into efficient model training.

SlimLM: Efficient small language models (SLMs) optimized for mobile document assistance, achieving comparable or superior performance to existing SLMs.

LLaMA-Mesh: Unifying 3D mesh generation with LLMs by directly representing meshes as text, enabling efficient text-to-3D conversion within a single model.

LLMs can dynamically adjust decoding temperature using Adaptive Decoding and Latent Preference Optimization, improving performance across creative and factual tasks.

Cut Cross-Entropy (CCE) dramatically reduces the memory footprint of training large language models by cleverly computing the cross-entropy loss without materializing the full logit matrix.

Sparse autoencoders fail to accurately decompose and interpret steering vectors due to distribution mismatch and the inability to handle negative feature projections; this paper identifies these issue…

CamemBERT 2.0: Two new French language models (CamemBERTav2 & CamemBERTv2) outperform predecessors by addressing temporal concept drift via larger, updated datasets and enhanced tokenization, demonstr…

Top-ησ: A novel LLM sampling method outperforms existing approaches by using a statistical threshold on pre-softmax logits, achieving higher accuracy while maintaining diversity, even at high temperat…

LLMs can now self-improve long-context reasoning via SEALONG, a novel method leveraging multiple model outputs and minimum Bayes risk scoring to enable effective supervised fine-tuning or preference o…

Feature-level constrained Preference Optimization (FPO) boosts LLM alignment efficiency and stability by using sparse autoencoders and feature-level constraints, achieving significant improvements ove…

Larger language models aren’t always better teachers for instruction tuning; a new metric, CAR, predicts teacher model effectiveness better than existing methods.

Chinese SimpleQA, a new benchmark, offers a comprehensive evaluation of the factuality of LLMs answering short questions in Chinese, exhibiting diversity, high quality, and ease of evaluation.

WEB-DREAMER uses LLMs as world models for safe and efficient web agent planning, achieving substantial performance gains over reactive baselines.

Contrary to common belief, toxicity reduction in language models isn’t simply achieved by dampening toxic neurons; it’s a complex balancing act across multiple neuron groups.