Name: Ali Behrouz | Titans: Learning to Memorize at Test Time
Start: 2025-03-26T18:30:00Z
End: 2025-03-26T19:30:00Z

**Title:** Titans: Learning to Memorize at Test Time
**Speaker:** Ali Behrouz (PhD, Cornell University)
**Paper:** [https://arxiv.org/abs/2501.00663](https://arxiv.org/abs/2501.00663)
**Abstract:** Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long-past information. We show that this neural memory has the advantage of fast parallelizable training. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, and time series tasks show that Titans are effective compared to baselines, while they can effectively scale to larger than 2M context window size in needle-in-haystack tasks.
**Speaker bio:** Ali Behrouz is a Ph.D. student in the Computer Science Department at Cornell University and a research intern at Google Research. His research spans topics from deep learning architectures to their applications in healthcare and neuroscience, and appeared at NeurIPS, ICML, KDD, WWW, CHIL, VLDB, ... conferences. His work has been featured with two Best Paper awards, a Best Paper Honorable Mention award, a Best Paper Award candidate, and oral and spotlight presentations.
**Agenda**:

* 18:25: Virtual doors open
* 18:30: Talk
* 19:10: Q&A session
* 19:30: Close

Apache Spark+AI London

London Machine Learning and Artificial Intelligence Talks

The organiser, Evolution AI, is an award-winning data extraction firm.

Evolution AI

Man Group

G-Research

ArcticDB

Capgemini

Technology

Apache

Cloud Computing

Hadoop

Big Data

Machine Learning

Data Mining

Apache Spark

Ali Behrouz | Titans: Learning to Memorize at Test Time

Artificial Intelligence

London

Software Development

Computer Science

Online event

Share

Apache Spark+AI London

Ali Behrouz | Titans: Learning to Memorize at Test Time

Apache Spark+AI London

Details

Sponsors

Members are also interested in