Deep Tech Meetup #7 - Evolution of Representations in the Transformer

Hello Bot-Enthusiasts,

This will be our seventh meetup with a strong focus on tech and research. The talk plus the Q&A will be ca. 60 min, so there will be plenty of time for discussions.

We seek to understand how the representations of individual tokens and the structure of the learned feature space evolve between layers in deep neural networks under different learning objectives. We focus on the Transformers for our analysis as they have been shown effective on various tasks, including machine translation (MT), standard left-to-right language models (LM) and masked language modeling (MLM). Previous work used black-box probing tasks to show that the representations learned by the Transformer differ significantly depending on the objective. In this work, we use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers and how this process depends on the choice of learning objective. For example, as you go from bottom to top layers, information about the past in left-to-right language models gets vanished and predictions about the future get formed. In contrast, for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation. The token identity then gets recreated at the top MLM layers.

Speaker:

Lena Voita:
Elena (Lena) Voita, is a Ph.D. candidate at the University of Amsterdam. She often visits the School of Informatics at the University of Edinburgh and is a part of the EdinburghNLP group. Besides that she is a research scientist at Yandex Research and is working on Natural Language Processing. Her research focuses on Neural Machine Translation, and she is working closely with the Yandex Translate team. Also she is teaching NLP at the Yandex School of Data Analysis.
More information about Lena you can find here: https://lena-voita.github.io/.

Agenda:

6:30-7:00pm Doors open & Networking + Pizza
7:00-8:00pm Talk + Q&A
8:00-8:30pm Networking & Beer

Where: Rasa Office, Schönhauser Allee 175, 10119, Berlin

Live stream: https://www.youtube.com/watch?v=h5N7sbAKBhA

See you soon,
The Rasa Team