Drew Jaegle | Perceivers: Towards General-Purpose Neural Network Architectures

Virtual London Machine Learning Meetup - 11.05.2022 @ 18:30

We would like to invite you to our next Virtual Machine Learning Meetup.

Agenda:

• 18:25: Virtual doors open
• 18:30: Talk
• 19:10: Q&A session
• 19:30: Close

Sponsor: Evolution AI - Intelligent data extraction from corporate and financial documents.

Title: Perceivers: Towards General-Purpose Neural Network Architectures (Drew Jaegle, Research Scientist at DeepMind)

Papers:

• Perceiver: https://arxiv.org/abs/2103.03206
• Perceiver IO: https://arxiv.org/abs/2107.14795
• Perceiver AR: https://arxiv.org/abs/2202.07765
• Hierarchical Perceiver: https://arxiv.org/abs/2202.10890

Abstract: A central goal of AI is to build systems that can flexibly process data from any domain, but current neural network architectures are designed to handle essentially one data configuration each.

This talk will cover Perceivers, a family of architectures that scale well to many kinds of high-dimensional data while making essentially no domain assumptions, which makes them easy to adapt and tune to a wide range of problems in many domains. Perceivers leverage an asymmetric attention mechanism to encode and decode data from a latent bottleneck. This mechanism allows Perceivers to handle data several orders of magnitude larger than can be used with Transformers, making it possible to apply them to real-world domains with much less preprocessing and special-casing. This family of architectures obtains performance comparable to or better than domain-specific architectures on a wide range of tasks.

Also included will be examples of recent work extending Perceivers to produce state of the art results on long-context generative modeling and to process extremely large data like megapixel-scale HD images.

Bio: Drew Jaegle is a Research Scientist at DeepMind in London. Previously, he worked on computer vision and computational neuroscience at the University of Pennsylvania, where he completed a PhD in the GRASP Lab with Kostas Daniilidis and a postdoc with Nicole Rust.