Theoretical Foundations of Deep Selective State-Space Models

Attention mechanisms have been the dominant approach in sequence-to-sequence modeling for several years. However, recent advancements in state-space models (SSMs) offer a compelling alternative, showing competitive performance and scalability advantages. A key challenge, however, is understanding the limitations of standard SSMs in tasks requiring input-dependent data selection, like content-based reasoning in text and genetics. This paper investigates this challenge and seeks to improve the design and performance of such models.

This research addresses the limitations of SSMs by providing a theoretical framework for the analysis of generalized selective SSMs. The authors leverage tools from rough path theory to fully characterize the expressive power of these models. They identify the input-dependent gating mechanism as the crucial design aspect responsible for superior performance. The findings not only explain the success of recent selective SSMs but also provide a solid foundation for developing future SSM variants, suggesting cross-channel interactions as key areas for future improvement. This rigorous theoretical analysis bridges the gap between theoretical understanding and practical applications, guiding the design of more effective and efficient deep learning models for sequential data.