Optimization

µP²: Layerwise perturbation scaling in SAM enables hyperparameter transfer and improved generalization in large models.

Deep learning’s learning rate warmup improves performance by allowing larger learning rates, pushing networks to better-conditioned loss landscape areas.

Adam’s superiority over SGD in Transformer training is explained by the ‘block heterogeneity’ of the Hessian matrix, highlighting the need for adaptive learning rates.

Metric backbone graph sparsification surprisingly preserves community structure, offering an efficient and robust method for analyzing large networks.

Weight decay’s role in modern deep learning is surprisingly multifaceted, impacting optimization dynamics rather than solely regularization, improving generalization and training stability.

Unlocking optimal neural network training: A narrow range of initially high learning rates, slightly above the convergence threshold, consistently yields superior generalization after fine-tuning.

This paper provides a tight characterization of inductive inference, proving it’s possible if and only if the hypothesis class is a countable union of online learnable classes, resolving a long-standi…

Planning is redefined as a distinct inference type within a variational framework, enabling efficient approximate planning in complex environments.

Diffusion guidance, a common generative modeling technique, is shown to not sample from its intended distribution; instead, it heavily biases samples towards the boundary of the conditional distributi…

This research introduces the first theoretical guarantees for warm-starting the celebrated Push-Relabel network flow algorithm, improving its speed using a predicted flow, while maintaining worst-case…

Unlocking the optimal variance estimation rate in compound decision theory under bounded means, this paper reveals a surprising (log log n/log n)² rate and introduces a rate-optimal cumulant-based est…

Researchers developed Spherical Convolutional Wasserstein Distance (SCWD) to more accurately validate climate models by considering spatial variability and local distributional differences.

Recommender systems, when influenced by both users and creators, inevitably polarize; however, prioritizing efficiency through methods like top-k truncation can surprisingly enhance diversity.

Recommendation algorithms on UGC platforms face a critical trade-off: prioritizing user satisfaction reduces creator engagement, jeopardizing long-term content diversity. This research introduces a ga…

Unrolled neural networks, trained via gradient descent, provably achieve optimal Bayesian inference for compressed sensing, surpassing prior-aware counterparts.

This paper reveals how large embedding dimensions and appropriate initialization guarantee convergence in Transformer training, highlighting Gaussian attention’s superior landscape over Softmax.

Adaptive gradient methods using AdaGrad stepsizes achieve optimal convergence rates for convex composite optimization problems, handling inexact oracles, acceleration, and variance reduction without n…

This paper introduces a novel universal online convex optimization algorithm needing only one projection per round, achieving optimal regret bounds for various function types, including general convex…

Repeated self-distillation significantly reduces excess risk in linear regression, achieving up to a ’d’ factor improvement over single-step methods.

A revolutionary ultrafast phylogenetic method outperforms protein language models in variant effect prediction by efficiently estimating amino acid substitution rates from massive datasets.