Revisiting Adam for Streaming Reinforcement Learning

Bucharest Deep Learning is back with another exciting session! Join us for a deep dive into Streaming Reinforcement Learning alongside Florin Gogianu, presenting recent research on making pure sequential learning highly competitive.

The Talk: Revisiting Adam for Streaming Reinforcement Learning

Learning from interactions as soon as observations are perceived, without explicitly storing them, promises simpler, more efficient, and adaptive algorithms. However, for over a decade, Deep RL has relied heavily on memory-intensive replay buffers or parallel sampling to tame learning instability.

This talk takes a step back to investigate the efficacy of established batch-RL algorithms, like DQN and C51, within the pure online streaming setting. Florin will show that when appropriately tuned, standard optimization techniques can be surprisingly effective. The presentation will unpack the "unreasonable effectiveness" of the Adam optimizer , demonstrating that two properties are essential for robust streaming performance: bounded objective derivatives and variance-adjusted weight updates.

Finally, the talk will introduce Adaptive Q(λ) (AQ(λ)). By combining eligibility traces with Adam's variance adaptation mechanism and a bounded error signal, this new algorithm surpasses existing streaming methods and approaches double the human baseline across a subset of 37 Atari games.

Logistics

• Date & Time: Tuesday, May 26 | 18:30 - 19:30
• Location: FMI New Building (Politehnica Business Tower)
• Address: Bulevardul Iuliu Maniu, nr. 15G, Etaj 5, Room 503