Titans: Long-Context Sequence Models with Test-Time Memorization

This month's meetup is about the Titans neural network architecture published by Google Research:

The goal of the architecture is to:

• scale transformer-based sequence models for tasks requiring very long context
• enable test-time memorization

This allows models to maintain long-term memory effectively while running—without the need for dedicated offline retraining.

We'll discuss the Titans architecture, with a focus on its implementation of the long-term memory module:

• how neural networks can act as memory vs. classical fixed-size recurrent states
• how this type of long-term memory compares to traditional RNNs, and why it enables models to learn new context on the fly

We will also take a look at the MIRAS framework, which provides a higher-level abstraction for designing sequence models.

The key idea of MIRAS is that sequence model architectures—from transformers to linear RNNs—can be viewed as complex associative memories under the hood.

We'll see how Titans fits into this framework of neural network design, and why the authors state that Titans and MIRAS can: "combine the speed of RNNs with the accuracy of transformers."

***

Follow-up from the last event
HOPE, the reference architecture for Nested Learning, is built on a modified version of the Titans architecture. At our last event, we didn’t have time to discuss many important details about how HOPE is built.
So, the March meetup will also act as a follow-up for anyone who missed some of the underlying architectural details last time.

=== ENTRY DETAILS ===

- QR code with entry information will be available soon, in the "Photos" section of this event page.
- Gate closes at 18:15 - no late entries.