**In person** Edoardo Ponti: Efficiency as an Inductive Bias for Language Models

**Only register here if you attend in person. Please use this link for online attendance: https://www.meetup.com/ucl-natural-language-processing-meetup/events/300762913/?isFirstPublish=true

We are delighted to have Edoardo Ponti from the University of Edinburgh giving a talk on "Efficiency as an Inductive Bias for Language Models".

Efficiency in Natural Language Processing is often hailed as a solution to democratise access to AI technology and to make it more environmentally sustainable. In this talk, I emphasise an additional and sometimes neglected advantage of efficiency: namely, providing an inductive bias for language use and acquisition closer to humans, where information-theoretic trade-offs shape the very structure of language.
In particular, I will explore how efficient designs in language models (a) may also act as inductive biases that improve their usefulness (b). For instance:
(1a) Jointly learning to model and tokenise language allows for merging spans of tokens in the intermediate layers of Transformers or in their key-value cache, which reduces time and memory requirements. (1b) In addition, this process also discovers possibly reusable and hierarchical abstractions (such as linguistic units) from raw data. What is more, this results in tokenization-free models that can integrate multiple modalities with different input spaces.
(2a) Learning parameter-efficient modules allows for fine-tuning LLMs with limited memory budgets. (2b) In addition, mixing these specialised modules through appropriate routing also leads to better generalisation. In particular, I will show how modules can be implemented as highly composable sparse adapters and how routing through modules can be learned automatically.
In conclusion, efficient designs of LLMs yield unexpected benefits, such as the ability to learn abstractions, dispose of tokenizers, and adapt fast to new tasks.

Bio: Edoardo M. Ponti is a Lecturer in Natural Language Processing at the University of Edinburgh and an Affiliated Lecturer at the University of Cambridge. Previously, he was a visiting postdoctoral scholar at Stanford University and a postdoctoral fellow at Mila and McGill University in Montreal. In 2021, he obtained a PhD in computational linguistics from the University of Cambridge, St John’s College. His main research foci are modular deep learning, sample-efficient learning, faithful text generation, computational typology and multilingual NLP. His research earned him a Google Research Faculty Award and 2 Best Paper Awards at EMNLP 2021 and RepL4NLP 2019. He is a (terrible) violinist, football and tennis player, and an aspiring practitioner of heroic viticulture.

The talk will be on the 1st floor at 90 High Holborn in the room called Function Space (on the left-hand side when you pass the door on the 1st floor). We strongly encourage people to **arrive on time**.

There will be **pizza** and socializing afterwards.