Large Language Models

This study reveals that gist token-based context compression in LLMs, while effective for some tasks, suffers from key failure patterns. The authors propose fine-grained autoencoding and segment-wise…

Long-context models surprisingly show that simple random sampling of examples is as effective as sophisticated methods for in-context learning, shifting the focus to efficient context utilization.

OpenRFT adapts generalist reasoning models for domain-specific tasks using reinforcement fine-tuning, overcoming data scarcity and lack of reasoning step data via question augmentation, synthesized re…

NILE framework significantly boosts LLM performance by aligning instruction-tuning datasets with pre-trained internal knowledge, achieving up to 68.5% gains.

ROBUSTFT tackles noisy data in LLM fine-tuning by using multi-expert noise detection and context-enhanced relabeling, significantly boosting model performance in noisy scenarios.

ReMoE: Revolutionizing Mixture-of-Experts with fully differentiable ReLU routing, achieving superior scalability and performance.

Boosting code generation accuracy, Outcome-Refining Process Supervision (ORPS) uses execution feedback and structured reasoning to refine code, achieving significant improvements across models and dat…

MixLLM achieves state-of-the-art LLM compression by using mixed-precision quantization between output features, improving accuracy and system efficiency.

M-ALERT, a new multilingual benchmark, reveals significant safety inconsistencies across languages in top LLMs.

Token-level editing prevents language model collapse from synthetic data by theoretically bounding test error and empirically improving model performance.

Fietje: an open-source, efficient Dutch language model outperforming larger models.

AceMath achieves state-of-the-art results in mathematical reasoning by introducing highly effective instruction-tuned models and reward models.

AI agents are tested in a simulated company, revealing their capability to automate tasks and shortcomings with complex workflows and interfaces.

First benchmark for RAG reward models reveals their limitations and the need for preference-aligned training.

Mix-LN boosts deep layer power in LLMs.

Auto-built benchmark with up-to-date knowledge ensures contamination-free LLM evaluation.

G-Pass@k & LiveMathBench: Evaluating the stability of LLMs.

Self-play method SPAR enhances LLMs instruction following abilities, beating GPT-4 on IFEval

SepLLM shrinks LLMs, speeding them up by over 50% without losing much accuracy.

Smaller is better: SLMs outperform LLMs in evolving complex & diverse instructions for AI training.