Generalization

Learned dynamical systems often fail to capture true physical behavior; this work introduces an ergodic theoretic approach to improve statistical accuracy by incorporating Jacobian information during …

This work systematically investigates the approximation properties of Transformer networks for sequence modeling, revealing the distinct roles of key components (self-attention, positional encoding, f…

This paper introduces a novel theoretical framework for robust machine learning under distribution shifts, offering learning rules and guarantees, highlighting the game-theoretic viewpoint of distribu…

Generative models can outperform their human trainers: A groundbreaking study shows how autoregressive transformers, trained on chess game data, can achieve higher game ratings than any of the human …

New topology-based complexity measures reliably predict deep learning model generalization, outperforming existing methods and offering practical computational efficiency.

Novel theoretical guarantees for Diffusion Flow Matching (DFM) models are established, bounding the KL divergence under mild assumptions on data and base distributions.

Deep learning’s sequence modeling is revolutionized by selective state-space models (SSMs)! This paper provides theoretical grounding for their superior performance, revealing the crucial role of gati…

Strong student models can learn from weaker teachers, even correcting errors and generalizing beyond the teacher’s expertise. This paper provides new theoretical bounds explaining this ‘weak-to-strong…

Hard attention transformers show surprisingly greater power when processing numerical data sequences, exceeding capabilities on string data; this advancement is theoretically analyzed via circuit comp…

Testably learning polynomial threshold functions efficiently, matching agnostic learning’s best guarantees, is achieved, solving a key problem in robust machine learning.

Deep ReLU networks trained with large, constant learning rates avoid overfitting in univariate regression due to minima stability, generalizing well even with noisy labels.

Sharper ASGD generalization bounds achieved by leveraging on-average model stability, even without Lipschitz and smoothness assumptions; validated with diverse machine learning models.

Sketchy Moment Matching (SkMM) is a fast and theoretically sound data selection method for deep learning finetuning. By controlling variance-bias tradeoffs in high dimensions, SkMM drastically reduces…

Deep learning’s generalization ability defies conventional wisdom; this paper proves that overfitting in deep neural networks is ’tempered’, neither catastrophic nor perfectly benign, for both minimal…

Position coupling, a novel method, enhances the length generalization ability of arithmetic Transformers by directly embedding task structures into positional encodings. This simple technique enables…

This paper introduces a novel technique to bound prediction risks in new domains using causal diagrams, enabling reliable AI performance guarantees.

Partially observing neural circuits during experiments can create misleading models, even if single neuron activity matches; researchers need better validation methods.

New PAC-Bayes oracle bound extends Cramér-Chernoff to unbounded losses, enabling exact parameter optimization and richer assumptions for tighter generalization bounds.

Ridgeless regression, surprisingly, generalizes well even with noisy data if dimension scales sub-polynomially with sample size.

High-dimensional spectral algorithms show saturation effects: Kernel Ridge Regression underperforms optimal algorithms like gradient flow when regression functions are very smooth.