16. LightOn AI Meetup: Convergence&Implicit Regularization of Feedback Alignment

LightOn is redefining computing for some of today’s largest software/hardware challenges in AI and HPC, such as extreme-scale foundation models. Our vision is to lower the barrier to entry of Transformative AI for small and large businesses, in a sustainable way. We gather experts, engineers, researchers, who are shaping the world of tomorrow. We are back for our 16th edition!

Our meetups will happen online in order to keep our attendees safe until further notice.

This event takes place at 18:00, Paris time (UTC+1)

A Zoom link will be provided prior to the meetup.

---
Agenda

18:00 – Introduction by LightOn

---

18:05 – Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
by Manuela Girotti, Assistant Professor at Saint Mary's University (Halifax, NS) and Research Member at Mathematical Science Research Institute (UC Berkeley, CA)

Abstract: We consider the Feedback Alignment algorithm, a bio-plausible alternative to backpropagation for training neural networks, and we analyze (1) convergence rates for deep linear networks and (2) incremental learning phenomena for shallow linear networks. Interestingly, depending on the initialization, the principal components of the model may be learned first (implicit regularization) or after (implicit anti-regularization) the negligible ones, thus affecting the effectiveness of the learning process.