Scalable Adaptive Computation for Iterative Generation

Please note: this event is in hybrid format, taking place in Science Park Lab42: L3.33 and streamed via Zoom: https://uva-live.zoom.us/j/81789020266

Speaker: Allan Jabri from Berkeley AI Research

Abstract: Progress in generative models of high-dimensional data such as images and video is contingent on scalable neural network architectures. We propose the Recurrent Interface Network (RIN), a neural net architecture that can allocate computation adaptively to the input according to the distribution of information, allowing it to scale to iterative generation of high-dimensional data. Hidden units of RINs are partitioned into the interface, which is locally connected to inputs, and latents, which are decoupled from inputs and can exchange information globally. The RIN block selectively reads from the interface into latents for high-capacity processing, with incremental updates written back to the interface. Stacking multiple blocks enables effective routing across local and global levels. While routing adds overhead, the cost can be amortized in recurrent computation settings where inputs change gradually while more global context persists, such as iterative generation using diffusion models. To leverage this structure, we propose a latent self-conditioning technique that "warm-starts" the latents at each iteration of the generation process. When applied to diffusion models operating directly on pixels, RINs yield state-of-the-art image and video generation without cascades or guidance, while being domain-agnostic and up to 10× more efficient compared to specialized 2D and 3D U-Nets.

Bio: Allan is a PhD student at Berkeley and a member of Berkeley AI Research, advised by Alexei A. Efros.
Currently, he is an intern at DeepMind in London. Previously, he was a student researcher at Google Brain Toronto.