Name: [Paper Reading]: The Path Not Taken: RLVR Provably Learns Off the Principals
Start: 2025-12-03T19:00:00-08:00
End: 2025-12-03T21:00:00-08:00

This week, we will walk through and discuss the paper: The Path Not Taken: RLVR Provably Learns Off the Principals [https://arxiv.org/pdf/2511.08567]

**Abstract of the Paper:**
Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR.

Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

We are a group of applied AI practitioners and enthusiasts who have formed a collective learning community. Every Wednesday evening at PM PST, we hold our research paper reading seminar covering an AI topic. One member carefully explains the paper, making it more accessible to a broader audience. Then, we follow this reading with a more informal discussion and socializing.
You are welcome to join this in person or over Zoom. **[SupportVectors](https://supportvectors.io/)**[ ](https://supportvectors.io/)is an AI training lab located in Fremont, CA, close to Tesla and easily accessible by road and BART. We follow the weekly sessions with snacks, soft drinks, and informal discussions.
If you want to attend by Zoom, the Zoom registration link will be visible once you RSVP. Note that we have had to change and add security to the Zoom link to prevent Zoom bombing.

Samantha Damaso

SupportVectors: Generative AI, LLMs, Machine Learning

SupportVectors

Technology

Neural Networks

Natural Language Processing

Artificial Intelligence

Machine Learning

Data Visualization

Data Science

Machine Intelligence

Deep Learning

Artificial Intelligence Applications

Artificial Intelligence Programming

Takeo Shibata

Tan Bui

Ganaraja NG

Herlene

Srikanth Tanniru

Mike Hillowalla

Technology Leader \| AI/Data Scientist \| Computer Scientist \| Educator \| Theoretical Particle Physicist

https://www.linkedin.com/in/asifqamar/

Asif Qamar

[Paper Reading]: The Path Not Taken: RLVR Provably Learns Off the Principals

Share

SupportVectors: Generative AI, LLMs, Machine Learning

[Paper Reading]: The Path Not Taken: RLVR Provably Learns Off the Principals

SupportVectors: Generative AI, LLMs, Machine Learning

Details

Members are also interested in