Large Language Models

MINI-SEQUENCE TRANSFORMER (MST) drastically reduces memory usage in LLM training by processing mini-sequences iteratively, enabling training with 12-24x longer sequences than conventional methods with…

MInference 1.0 accelerates LLM pre-filling via dynamic sparse attention, achieving up to 10x speedup on an A100 GPU while maintaining accuracy.

MindMerger efficiently boosts LLM reasoning in non-English languages by merging LLMs with external multilingual language understanding capabilities, achieving significant accuracy improvements, especi…

LLMs’ spatial reasoning abilities are boosted by visualizing their thought processes via ‘Visualization-of-Thought’ prompting, significantly improving performance on navigation and tiling tasks.

LLMs struggle with graph recall, exhibiting biases like favoring triangles and underperforming compared to humans; advanced models show striking domain dependence.

MICROADAM: A new Adam optimizer variant dramatically cuts memory usage for training large language models without compromising accuracy or provable convergence.

MetaLA: Unified optimal linear approximation to softmax attention map, achieving linear complexity and surpassing existing models in various benchmarks.

LLMs gain math skills via prompt-guided skill labeling and exemplar selection, significantly boosting accuracy.

MetaAligner: a novel, policy-agnostic, and generalizable method for efficiently aligning LLMs to multiple objectives, even unseen ones, achieving significant and balanced improvements while saving up …

Mesa-Extrapolation enhances LLM extrapolation using a novel weave position encoding method, boosting performance while significantly reducing memory and inference time.

MemoryFormer drastically cuts large language model computation by replacing fully-connected layers with memory-efficient hashing, enabling faster and more scalable AI.

Online Subspace Descent: a novel memory-efficient LLM training algorithm guaranteed to converge, closing the performance gap with full-rank methods.

SPV-MIA, a novel membership inference attack, significantly improves the accuracy of identifying training data in fine-tuned LLMs by using self-prompt calibration and probabilistic variation.

MEGALODON: A new neural architecture for LLMs, enabling unlimited context length with improved efficiency and accuracy.

Boosting LLMs’ abstract reasoning via ‘Meaningful Learning’: A new dataset and learning paradigm significantly enhance LLMs’ capacity for abstract reasoning, moving beyond simple memorization.

MatFormer: Train one universal model, extract hundreds of accurate submodels for elastic inference!

MaskLLM learns efficient semi-structured sparsity in LLMs via end-to-end training, achieving significant speedup and memory reduction without sacrificing performance.

Long-context attacks easily manipulate LLMs by feeding hundreds of harmful examples, highlighting a critical vulnerability amplified by larger context windows.

Scaling up in-context learning using thousands of examples significantly boosts Large Language Model (LLM) performance, particularly for complex tasks. Novel training methods mitigate reliance on hum…

MAmmoTH2: Harvesting 10M web instructions for enhanced LLM reasoning!