On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

Existing research on Adam optimizer’s convergence often relies on restrictive assumptions, such as bounded gradients and specific noise models. This limits the applicability of these results to real-world scenarios. The non-convex smooth setting is further complicated by the adaptive step sizes used in Adam, which can interact complexly with the noise. This makes analysis challenging, and many existing studies provide weak convergence guarantees or require modifications to Adam’s algorithm.

This paper tackles these challenges head-on. It analyzes Adam’s convergence under relaxed assumptions, including unbounded gradients and a general noise model that encompasses various noise types commonly found in practice. Using novel techniques, the authors prove high probability convergence results for Adam, achieving a rate that matches the theoretical lower bound of stochastic first-order methods. The findings extend to a more practical generalized smooth condition, allowing the algorithm to handle a wider range of objective functions encountered in applications. These results are significant because they provide stronger theoretical backing for Adam and improve its practical applicability.