Efficient Language Models: finding your optimal architecture
Details
Meghana Ravikumar from SigOpt will join us to talk about finding the architecture for your language models.
Summary:
Most pretrained language models are too large for many use cases, especially if they need to run on devices like phones, and they are not adapted to your domain. So, how can you distill a language model to a size that makes them easier to deploy, while at the same time adapting them to your task?
In this talk, we will explore how to reduce the size of BERT while retaining its capacity, using a Question Answering task as a worked example. We will cover fine-tuning and distillation using a teacher-student model. We will show how multimetric hyperparameter optimization can find the optimal architecture for the student model. Finally, we explore the trade-offs of our hyperparameter decisions to draw insights for our student model’s architecture.
Bio:
Meghana is a Machine Learning Engineer at SigOpt. She has worked with machine learning in academia and in industry and previously worked in biotech, employing NLP to mine and classify biomedical literature.
We will share details of how to join this online presentation closer to the date!
