Large Language Models

RDS+ is the unsung hero for scaling instruction tuning data selection!

Transformers get forgetful! This paper introduces the Forgetting Transformer (FoX), incorporating a forget gate into the attention mechanism for improved sequence modeling.

Shakti SLMs: Fine-tuning compact language models for efficient, domain-specific AI on edge devices.

CROWDSELECT boosts instruction tuning by cleverly selecting synthetic data using multi-LLM wisdom, enhancing model performance across diverse tasks.

CodeArena: Collective evaluation for LLM code generation.

DuoDecoding: Accelerating LLM inference by strategically deploying draft & target models on CPU & GPU for parallel decoding and dynamic drafting.

Babel: An open multilingual LLM supports over 90% of global speakers, filling the language coverage gap and setting new performance standards.

LongRoPE2: Extends LLM context windows while preserving performance and reducing training costs!

NeoBERT: A new encoder that enhances bidirectional language understanding with cutting-edge architecture, data, and training, achieving SOTA results with only 250M parameters.

CoD: LLMs think faster by writing less! A novel prompting strategy cuts costs and latency while maintaining reasoning accuracy.

GOAT: Adaptively boosts LoRA with SVD & MoE alignment, closing the gap with Full FT.

Test-time scaling isn’t a universal solve-all for multilingual math reasoning, unlike pre-training scaling, shows MCLM benchmark.

LLMs: 03-mini achieves superior accuracy without longer reasoning chains, suggesting ’thinking harder’ matters more than ’thinking longer'.

LightThinker: LLMs dynamically compress intermediate steps, reducing memory & boosting reasoning efficiency without sacrificing accuracy.

SURVEYX automates academic survey generation, enhancing content and citation quality.

Packing new knowledge into LoRA adapters can harm LLMs! A delicate balance is needed to prevent performance decline.

LLMs have ‘Temporal Heads’ that process time-specific facts!

LORAM: Train small, infer large LLMs by memory-efficient LoRA training. Enables 70B parameter model training on a 20G HBM GPU, replacing A100-80G. Reduces parameter storage cost by 15.81x.

SIFT: Grounds LLM reasoning with ‘Stickers’ to highlight context and improve accuracy without extra training.

REFIND: Detects LLM hallucinations by directly leveraging retrieved documents, using a novel Context Sensitivity Ratio.