← Back to stories
AI Labs

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent…

You're reading a preview. The full article is published by Apple Machine Learning Research on their website.

Read the full story on Apple Machine Learning Research