AI Theory

µP²: Layerwise perturbation scaling in SAM enables hyperparameter transfer and improved generalization in large models.

Zipper: A novel statistical device resolves the degeneracy issue in algorithm-agnostic inference, enabling reliable goodness-of-fit tests with enhanced power.

Zeroth-Order Diffusion Monte Carlo (ZOD-MC) efficiently samples from non-log-concave distributions using only zeroth-order queries, overcoming metastability issues and outperforming state-of-the-art s…

Wide two-layer neural networks can generalize well from mislabeled adversarial examples because adversarial perturbations surprisingly contain sufficient class-specific features.

Deep learning’s learning rate warmup improves performance by allowing larger learning rates, pushing networks to better-conditioned loss landscape areas.

Adam’s superiority over SGD in Transformer training is explained by the ‘block heterogeneity’ of the Hessian matrix, highlighting the need for adaptive learning rates.

Metric backbone graph sparsification surprisingly preserves community structure, offering an efficient and robust method for analyzing large networks.

Weight decay’s role in modern deep learning is surprisingly multifaceted, impacting optimization dynamics rather than solely regularization, improving generalization and training stability.

Unlocking optimal neural network training: A narrow range of initially high learning rates, slightly above the convergence threshold, consistently yields superior generalization after fine-tuning.

CARED: a novel causal action recommendation model improves policy learning by collaborating with human experts and mitigating hidden confounding in observational data.

Multicalibration post-processing isn’t always necessary; models often implicitly achieve it, especially calibrated ones. For uncalibrated models, though, it significantly improves fairness.

This paper provides a tight characterization of inductive inference, proving it’s possible if and only if the hypothesis class is a countable union of online learnable classes, resolving a long-standi…

This paper introduces a novel, task-agnostic method for ranking embedding models using information sufficiency, a concept derived from communication theory and statistical experiments comparison, demo…

Learned dynamical systems often fail to capture true physical behavior; this work introduces an ergodic theoretic approach to improve statistical accuracy by incorporating Jacobian information during …

Planning is redefined as a distinct inference type within a variational framework, enabling efficient approximate planning in complex environments.

Researchers developed RUM, a refined unlearning meta-algorithm, that significantly improves existing unlearning methods by strategically refining forget sets and employing appropriate unlearning algor…

Tri-Hom disentangles graph homophily into label, structural, and feature aspects, providing a more comprehensive and accurate metric for predicting GNN performance.

Diffusion guidance, a common generative modeling technique, is shown to not sample from its intended distribution; instead, it heavily biases samples towards the boundary of the conditional distributi…

Using tropical geometry, researchers reveal that ReLU-activated message-passing GNNs learn continuous piecewise linear functions, highlighting their expressivity limits and paving the way for enhanced…

This paper introduces r-lWL, a new graph isomorphism test hierarchy that surpasses the limitations of the Weisfeiler-Leman test by counting cycles up to length r+2, and its GNN counterpart, r-lMPNN, w…