Alex and Cosmin go over Fantastic Pretraining Optimizers and Where to Find Them
Details
Fantastic Pretraining Optimizers and Where to Find Them
This Tuesday Alex and Cosmin will cover "Fantastic Pretraining Optimizers and Where to Find Them". The paper is practical, benchmarking Muon and alternatives. It's a good opportunity for us to catch up on optimization improvements.
Resources
video tutorial of muon https://www.youtube.com/watch?v=bO5nvE289ec
fpowtf - https://arxiv.org/abs/2509.02046
fpowtf 2 - https://arxiv.org/abs/2606.16899
Related topics
Artificial Intelligence
Deep Learning
Machine Intelligence
Machine Learning
Neural Networks