Large Language Models

LLaMA-Mesh: Unifying 3D mesh generation with LLMs by directly representing meshes as text, enabling efficient text-to-3D conversion within a single model.

Cut Cross-Entropy (CCE) dramatically reduces the memory footprint of training large language models by cleverly computing the cross-entropy loss without materializing the full logit matrix.

Sparse autoencoders fail to accurately decompose and interpret steering vectors due to distribution mismatch and the inability to handle negative feature projections; this paper identifies these issue…

CamemBERT 2.0: Two new French language models (CamemBERTav2 & CamemBERTv2) outperform predecessors by addressing temporal concept drift via larger, updated datasets and enhanced tokenization, demonstr…

LLMs can now self-improve long-context reasoning via SEALONG, a novel method leveraging multiple model outputs and minimum Bayes risk scoring to enable effective supervised fine-tuning or preference o…

Feature-level constrained Preference Optimization (FPO) boosts LLM alignment efficiency and stability by using sparse autoencoders and feature-level constraints, achieving significant improvements ove…

Larger language models aren’t always better teachers for instruction tuning; a new metric, CAR, predicts teacher model effectiveness better than existing methods.

Chinese SimpleQA, a new benchmark, offers a comprehensive evaluation of the factuality of LLMs answering short questions in Chinese, exhibiting diversity, high quality, and ease of evaluation.

Contrary to common belief, toxicity reduction in language models isn’t simply achieved by dampening toxic neurons; it’s a complex balancing act across multiple neuron groups.

IOPO empowers LLMs to master complex instructions via input-output preference optimization, boasting significant performance gains on a new benchmark, TRACE.

Golden Touchstone, a new bilingual benchmark, comprehensively evaluates financial LLMs across eight tasks, revealing model strengths and weaknesses and advancing FinLLM research.

Boost large language model training speed by 51% with Vocabulary Parallelism, a novel technique that balances computation and memory usage across pipeline stages.

OpenCoder, a top-tier open-source code LLM, is introduced, providing not only model weights and code but also reproducible training data, data processing pipelines, and training protocols, enabling co…

Can LLMs effectively handle information spread across vast, almost million-scale datasets? This research investigates this question by evaluating 17 LLMs on novel ‘needle threading’ tasks. These task…

Researchers developed Hardware and Software Platform Inference (HSPI) to identify the underlying GPU and software stack used to serve LLMs, enhancing transparency in the industry.

DELIFT: Data Efficient Language Model Instruction Fine-Tuning, drastically reduces the data needed for effective LLM fine-tuning without sacrificing performance.

BitNet a4.8 achieves comparable performance to existing 1-bit LLMs, but with significantly faster inference, by using a hybrid quantization and sparsification strategy for 4-bit activations.

Zebra-Llama, a context-aware LLM, democratizes rare disease knowledge by providing highly precise, context-rich information about Ehlers-Danlos Syndrome, significantly improving diagnostic support.

WEBRL: A self-evolving online curriculum reinforcement learning framework empowers open LLMs to excel as high-performing web agents, surpassing proprietary models.

Researchers discovered predictable scaling laws for activation sparsity in LLMs, showing how data, architecture, and model size influence sparsity, paving the way for more efficient and interpretable …