Natural Language Processing

LLMs struggle with positional bias and lack transparency when summarizing long contexts. This paper introduces SUnsET dataset and fine-tuning methods to improve unstructured evidence citation and summ…

Packing new knowledge into LoRA adapters can harm LLMs! A delicate balance is needed to prevent performance decline.

LLMs have ‘Temporal Heads’ that process time-specific facts!

LORAM: Train small, infer large LLMs by memory-efficient LoRA training. Enables 70B parameter model training on a 20G HBM GPU, replacing A100-80G. Reduces parameter storage cost by 15.81x.

REFIND: Detects LLM hallucinations by directly leveraging retrieved documents, using a novel Context Sensitivity Ratio.

MoM: Enhancing linear sequence modeling via mixture-of-memories for improved recall and reduced memory interference.

LongPO: Self-evolve LLMs to excel in long contexts via short-to-long preference optimization, boosting performance without sacrificing short-context skills.

Test-time scaling + confidence = better QA!

CRAW4LLM: Efficiently crawls web pages for LLM pretraining by prioritizing influence scores, boosting data quality & cutting crawling waste.

Autellix: Efficient LLM Serving for Agents

SafeRoute efficiently enhances LLM safety by adaptively using smaller and larger safety guard models, maximizing accuracy while minimizing costs.

Decomposed Reward Models (DRMs) extract diverse human preferences from binary comparisons using PCA, enabling flexible and interpretable LLM alignment.

Perovskite-LLM: a new knowledge-enhanced system boosts perovskite solar cell research by integrating a domain-specific knowledge graph, high-quality datasets, and specialized LLMs for superior knowled…

PAFT dynamically adjusts prompts during LLM fine-tuning, improving model robustness and generalization across diverse prompts without sacrificing performance or efficiency.

Multilingual LLMs Hallucinate! This study measures hallucination across 30 languages.

HEADINFER achieves memory-efficient LLM inference by cleverly offloading key-value cache to the CPU, enabling 4 million token inference on a single consumer GPU.

Crowd-based comparative evaluation significantly boosts LLM-as-a-judge accuracy by using crowd responses to expose deeper details, resulting in more reliable and efficient auto-evaluation.

LLMs can losslessly compress 1568 tokens into a single vector, surpassing prior methods by two orders of magnitude.

SYSGEN: A novel pipeline generates effective system messages for LLMs using open-source models, improving model responses and addressing data scarcity in supervised fine-tuning.

SAFE-SQL boosts Text-to-SQL accuracy by intelligently generating and filtering self-augmented examples for in-context learning, surpassing existing methods in challenging scenarios.