Generalization Bounds via Conditional $f$-Information

Machine learning research heavily relies on understanding generalization—how well a model trained on a dataset performs on unseen data. Information-theoretic approaches, particularly using mutual information (MI), have shown promise but suffer from limitations, such as producing unbounded measures that don’t reflect true generalization error. Existing MI-based bounds often involve complex techniques that might not be easily applicable to various models and loss functions. Prior attempts to resolve these issues involved tightening existing bounds or using variants of mutual information, but they failed to achieve an accurate generalization measure for several cases.

This work introduces a novel framework based on conditional f-information, a broader concept than MI. The authors develop a generic method for deriving generalization bounds using this framework, applicable to diverse loss functions. This method avoids previous limitations by carefully selecting the measurable function to eliminate the problematic cumulant-generating function from the variational formula. Importantly, the derived bounds recover many prior results while enhancing our understanding of their limitations. Empirical evidence showcases the improvement of the new bounds against earlier approaches.