#23 AI Series: DeepMind - F. Barbero

We are back! Get ready for the BLISS AI Speaker Series Winter 2025/26! NOTE: We changed rooms and will now be in C130!

We are excited to feature Federico Barbero, who is currently at DeepMind in London and PhD Student at University of Oxford and will discuss "Why do LLMs struggle with Long Context?", lasting approximately 45 minutes. After the talk, seize the opportunity to connect with fellow AI enthusiasts to share ideas and questions while enjoying free drinks and pizza. Door close by 7.15pm, so please come early! Also, "attend"ing (RSVP) here on Meetup is strictly necessary to be guaranteed entry.
Please note that Meetup has recently been quite keen on promoting its Plus program. However, you are not obligated to purchase it, as both our events and the platform remain free.

Who is this event for?
This event is open to everyone interested in state-of-the-art AI research. We especially design it for students, PhD candidates, academic researchers, and industry professionals with a research focus in machine learning.

Abstract: There is great interest in scaling the number of tokens that LLMs can efficiently and effectively ingest, a problem that is notoriously difficult. Training LLMs on a smaller context and hoping that they generalize well to much longer contexts has largely proven to be ineffective. In this talk, I will go over our work that aims to understand the failure points in modern LLM architectures. In particular, I will discuss dispersion in the softmax layers, generalization issues related to positional encodings, and smoothing effects that occur in the representations. Understanding these issues has proven to be fruitful, with related ideas now already being part of frontier models such as LLaMa 4. The talk is intended to be broadly accessible, but a basic understanding of the Transformer architectures used in modern LLMs will be helpful.

Bio: Federico is a PhD student in Computer Science at the University of Oxford (Trinity College), supervised by Michael Bronstein, where he works on Geometric Deep Learning with a broader interest in ML security, robustness, and privacy. He is currently at Google DeepMind in London, collaborating with Petar Veličković on Algorithmic Reasoning. In 2023, he interned at Microsoft Research Amsterdam with the AI4Science team on protein sampling. Before that, he completed an MPhil in Machine Learning and Machine Intelligence at the University of Cambridge (King’s College), supervised by Pietro Liò and Cristian Bodnar, where he worked on Topological Deep Learning.

We are BLISS e.V., the AI organization in Berlin that connects like-minded individuals who share great interest and passion for the field of machine learning. This winter 2025/26, we will, again, host an exciting speaker series on site in Berlin, featuring excellent researchers from Tübingen AI Center, DeepMind, Microsoft, King's College London, cohere, and ETH Zürich.
Website: https://bliss.berlin
Youtube: https://www.youtube.com/@bliss.ev.berlin

Disclaimer: By attending this event you agree to be photographed.