Introduction to Automatic Differentiation from a Machine Learning Perspective

Speaker: Barak Pearlmutter, PhD (Head of Brain and Computation Lab (http://www.bcl.hamilton.ie/), Maynooth University (http://www.nuim.ie/))

Automatic Differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.).

By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

AD is used a lot in more modern statistical techniques such as deep learning and Markov Chain Monte Carlo, as it avoids issues with precision that occur when using more traditional techniques such as finite difference.