Paper 59: Backprop as Functor

Rongmin Lu will take us through the following paper:

Backprop as Functor: A compositional perspective on supervised learning
Brendan Fong, David Spivak, Rémy Tuyéras

https://arxiv.org/abs/1711.10455v3

Abstract

A supervised learning algorithm searches over a set of functions A → B parametrised by a space P to find the best approximation to some ideal function f : A → B. It does this by taking examples (a, f (a)) ∈ A × B, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent — with respect to a fixed step size and an error function satisfying a certain property — defines a monoidal functor from a category of parametrised functions to this category of update rules. A key contribution is the notion of request function. This provides a structural perspective on backpropagation, giving a broad generalisation of neural networks and linking it with structures from bidirectional programming and open games.