Natural Language Processing

Mesa-Extrapolation enhances LLM extrapolation using a novel weave position encoding method, boosting performance while significantly reducing memory and inference time.

MemoryFormer drastically cuts large language model computation by replacing fully-connected layers with memory-efficient hashing, enabling faster and more scalable AI.

Online Subspace Descent: a novel memory-efficient LLM training algorithm guaranteed to converge, closing the performance gap with full-rank methods.

SPV-MIA, a novel membership inference attack, significantly improves the accuracy of identifying training data in fine-tuned LLMs by using self-prompt calibration and probabilistic variation.

MEGALODON: A new neural architecture for LLMs, enabling unlimited context length with improved efficiency and accuracy.

MEDIQ benchmark revolutionizes LLM evaluation by shifting from static to interactive clinical reasoning, revealing LLMs’ struggles with proactive information-seeking and highlighting the importance of…

New metrics and p-annealing improve sparse autoencoder training for better language model interpretability.

Boosting LLMs’ abstract reasoning via ‘Meaningful Learning’: A new dataset and learning paradigm significantly enhance LLMs’ capacity for abstract reasoning, moving beyond simple memorization.

MatFormer: Train one universal model, extract hundreds of accurate submodels for elastic inference!

Masked hard-attention transformers, with strict masking, precisely capture star-free languages, matching the expressive power of linear temporal logic.

Long-context attacks easily manipulate LLMs by feeding hundreds of harmful examples, highlighting a critical vulnerability amplified by larger context windows.

MAmmoTH2: Harvesting 10M web instructions for enhanced LLM reasoning!

FILM-7B, trained with Information-Intensive (IN2) training, significantly overcomes the ’lost-in-the-middle’ problem in long-context LLMs, enabling robust information retrieval from all context positi…

MagR: a novel preprocessing technique boosts post-training quantization of LLMs by reducing weight magnitudes without inference overhead, achieving state-of-the-art performance.

MAGNET, a novel adaptive gradient-based tokenization method, tackles multilingual language model bias by employing language-specific boundary predictors to achieve equitable segmentation across divers…

MAGIS: A novel LLM-based multi-agent framework significantly boosts GitHub issue resolution by leveraging agent collaboration for planning and coding, achieving an eight-fold performance increase comp…

Multi-Agent System for Condition Mining (MACM) dramatically boosts large language model accuracy in complex math problem-solving, exceeding existing methods by achieving higher accuracy and better gen…

LT-Defense: a searching-free backdoor defense for language models leveraging the long-tailed effect of poisoned data. It achieves 98% accuracy across 1440 models with less than 1% time cost of existin…

LSH-MoE accelerates Mixture-of-Experts training by 1.28x-2.2x via Locality-Sensitive Hashing, significantly reducing communication costs.

LoRA-GA: A novel initialization method dramatically speeds up low-rank adaptation (LoRA) for LLMs, achieving convergence rates comparable to full fine-tuning while improving performance.