Skip to content

11. LightOn AI Meetup: Reservoir Transformers

Photo of
Hosted By
Iacopo P. and 2 others
11. LightOn AI Meetup: Reservoir Transformers


LightOn is a DeepTech company creating hardware for new AI applications requiring a massive amount of data (molecular dynamics, fraud detection, conversational AI ...). We gather experts, engineers, researchers, who are shaping the world of tomorrow. We are back for our 11th edition!

Our meetups will happen online in order to keep our attendees safe until further notice.

This event takes place at 18:00, Paris time (GMT +02:00)

A Zoom link will be provided prior to the meetup.


18:00 – Introduction by LightOn


18:10 – "Reservoir Transformers"
by Douwe Kiela, Research Scientist at Facebook AI Research

Abstract: I will talk about our recent work on reservoir transformers. We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation, language modelling, and masked language model pretraining tasks.

18:30 - Q&A

During the event, you can share content using #LightOn & @LightOnIO

After the meet-up:

  • the slides will be available on our Medium page:
  • the video of the presentation will be available on our Youtube Channel:
Online event
This event has passed