Machine Learning Papers - Reading and Discussion #2

◾ About
In this weekly series of remote online sessions, we will read and discuss a recent machine learning or natural language processing paper. This will serve as an inclusive, open, and fun space to interact with students, experts, researchers, and practitioners. It’s an open discussion and the agenda is as follows:

• 🕐 45 minutes of paper reading
• 🕐 75 minutes of paper discussion

In this session we will continue from where we left off in the first session.
We will spend a few minutes reading and taking notes and then spend the majority of the session having an in-depth discussion about the paper.

◾ Join Zoom Meeting: https://us02web.zoom.us/j/89574991955

◾ Slack channel: #paper_reading_t5

◾ Past notes: https://github.com/dair-ai/ml-nlp-paper-discussions

Make sure to add the event to your calendar to get a notification before the event starts!

We recommend you to read the paper in advance but it’s not necessary. To learn more about this event and get access to all the notes, questions, and discussions, join our slack group here: https://join.slack.com/t/dairai/shared_invite/zt-dv2dwzj7-F9HT047jIGkunNKv88lQ~g

◾ Selected Paper for Session # 2

Title: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Authors: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

Link: https://arxiv.org/abs/1910.10683

◾ About dair.ai

Website: https://dair.ai/
GitHub: https://github.com/dair-ai
Twitter: https://twitter.com/dair_ai
Newsletter: https://dair.ai/newsletter/
Slack: https://join.slack.com/t/dairai/shared_invite/zt-dv2dwzj7-F9HT047jIGkunNKv88lQ~g

◾ Code of Conduct: https://github.com/dair-ai/dair-ai.github.io/blob/master/CODE_OF_CONDUCT.md