Rob and Cosmin go over "Full Bandwidth Transformer"

Full Bandwidth Transformer

This Tuesday we're going over "Full Bandwidth Transformer" from MSR https://arxiv.org/abs/2608.08888

Resources

twitter thread by author John Langford

• https://x.com/JohnCLangford/status/2095849181919261168

he speculates it might be relevant to the new rumored astra arch from openai

Other relevant papers

Same MSR group, rnn/latent prediction ideas:

• NextLat (2025) https://arxiv.org/abs/2511.05963 - an author presented it at Yacine's pod recently https://www.youtube.com/watch?v=coyYxJ26hlQ
• followup - hierarchical NextLat https://arxiv.org/abs/2608.05806
• Almost Free State Prediction Separation https://arxiv.org/abs/2609.03807

Google:

• recurrence and strong benchmark wins by Recirculation paper (got 15% win on gsm8k) https://arxiv.org/abs/2608.17981

MIT:

• Pretraining Recurrent Nets Without Recurrence https://arxiv.org/abs/2606.06479