Reinforcement Learning: Building an AlphaZero Training Pipeline

Inspired by Chapter 9.8.3 in Multi-Agent Reinforcement Learning: Foundations and Modern Approaches which introduces the AlphaZero algorithm, we will continue with a deeper dive into how to apply the algorithm and more modern extensions of it in a practical training pipeline. The final algorithm will be based on the following paper:
Danihelka, I., Guez, A., Schrittwieser, J., & Silver, D. (2022). Policy Improvement by Planning with Gumbel (ICLR 2022). https://openreview.net/forum?id=bERaNdoegnO
We will use an extended tic-tac-toe style game as a test environment and show how a single execution of Monte Carlo search improves upon an existing policy. Then we will show how the tree search can be used to build a dataset for training neural networks with a version of policy improvement. We will benchmark the data generation and build the training update functions in order to compare the performance. There are many hyperparameters to consider when building the training process including buffer size, minibatch steps per generation, resource allocation between search and training, etc... We will test different instantiations of these factors when building a candidate training pipeline and then test it in the MDP environment. We can also scale the difficulty of the environment as a way to test if these factors are robust to problem complexity.
As usual you can find below links to the textbook, previous chapter notes, slides, and recordings of some of the previous meetings.
Meetup Links:
Recordings of Previous RL Meetings
Recordings of Previous MARL Meetings
Short RL Tutorials
My exercise solutions and chapter notes for Sutton-Barto
My MARL repository
Kickoff Slides which contain other links
MARL Kickoff Slides
MARL Links:
Multi-Agent Reinforcement Learning: Foundations and Modern Approaches
MARL Summer Course Videos
MARL Slides
Sutton and Barto Links:
Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto
Video lectures from a similar course