Transformers: The Artistry of Sparsity

Transformers are everywhere, and it seems like every week we have a few new flavors of them to excel at our NLP tasks. However, to reach the next level we must solve the problem of scaling attention mechanisms. 2020 was focused on trying to intelligently cut down on the quadratically-scaling attention mechanism operation (plus that whole virus thing), so that we could begin to tackle NLP tasks on longer documents.

In this talk, Rob will review much of this literature on enhanced transformer models, focusing on sparsity and structures around compressed memory, and give a few tips on how this can be used in industry and your own research.