Transformers

The Transformer architecture has revolutionized in the last couple of years the way neural networks are regarded and used through fields like natural language processing and computer vision. With the latest models in NLP, I (Bogdan Musat) dare to say we are close to solving the old Turing imitation test.
The bitter lesson however is that Transformers scale with data and model complexity. The current state-of-the-art models can reach up to 540b neural connections and are extremely data-hungry, which is why only a few labs in the world can afford to scale to such complexities.
The question is then how can we make Transformers more affordable to everyone and even more important how to deploy them on the edge and make use of their true potential. We should strive to find the shortest computer program that can solve the task at hand. The length of this shortest computer program is also known in algorithmic information theory as Kolmogorov complexity.
To search for such a solution, we can use techniques like pruning and quantization in combination with heuristic approaches or automated machine learning frameworks. This presentation will cover the latest breakthroughs in the field of AI that are using Transformers as a core technology and then some solutions from neural network optimization and deployment which could make Transformers more affordable to everyone.

***

Bogdan Musat, the meeting’s speaker, is an Applied Scientist at Amazon Ring, where he develops AI-based technologies for smart home security devices that make neighborhoods safer. He is also a Ph.D. student in the field of Artificial Intelligence.

***

This is a joint meetup with Amazon Devices Meetup initiative ran by Cowork Timisoara and Amazon Development Center.