Building Deepseek v3 from scratch

Join us for an in-depth, hands-on architectural breakdown and live coding walkthrough of DeepSeek-V3. We will go beyond high-level summaries to examine the exact paper mathematics, key engineering innovations, and complete PyTorch implementation. Whether you want to understand how DeepSeek handles massive context windows efficiently or how its Mixture-of-Experts (MoE) dynamic load balancing works, this session covers every component line-by-line.

Monika will walk through the entire session. A little about Monika, She's a GenAI developer with experience in building AI Agents for Cloud-based business automation solutions, Winner of 3 All India AI hackathons.

Key Concepts Covered

• Attention & KV Cache Bottleneck: Standard Query-Key-Value (QKV) dot-product attention mechanics and why storing full KV representations creates critical memory and bandwidth bottlenecks in long-context LLMs.
• Multi-Head Latent Attention (MLA):
• Low-Rank Compression: Compressing high-dimensional token embeddings (~7,000+ dims) into compact latent vectors (~500–576 dims) to dramatically shrink the KV cache footprint.
• Decoupled RoPE: Splitting un-rotated semantic latent heads from rotated positional heads (`rope_head_dim`) to retain position awareness without cache inflation.
• Weight Absorption: Mathematically absorbing key/value projection weights into query projections and output matrices (WO​) to compute attention directly from latent vectors.
• RoPE Scaling & Softmax Adjustments (mscale​):
• Context Extension: Scaling rotation frequencies by a `rope_factor` to expand context capacity (e.g., 100k to 200k tokens).
• Attention Sharpening: Applying logarithmic Softmax scale adjustments (mscale2​) to counteract uniform "diffusion" caused by slower rotations and restore optimal attention confidence.
• DeepSeek Mixture-of-Experts (MoE) & Load Balancing:
• Shared vs. Routed Experts: Combining permanent Shared Experts (universal language/grammar knowledge) with specialized Routed Experts (domain-specific routing).
• Hierarchical Grouped Top-K Gating: Efficient token routing via grouped candidate selection using Softmax/Sigmoid routing.
• Aux-Loss-Free Balancing: Replacing auxiliary-loss networks with dynamic per-expert bias terms (γ)—penalizing over-utilized experts and boosting under-utilized ones without hurting primary training loss.
• Distributed GPU Parallelization:
• Column Parallel: Splitting output feature dimensions across GPUs.
• Row Parallel: Splitting input feature dimensions across GPUs and aggregating partial sums.

PyTorch Hands-On Code Walkthrough

We will implement the following PyTorch modules step-by-step:

• MLA Class: Down-projections, latent cache storage, decoupled RoPE splitting, and weight absorption tricks.
• Gate & MoE Classes: Grouped top-K selection, dynamic bias adjustments, affinity normalization, and expert routing.
• MLP / Expert Modules: SwiGLU gating mechanisms (W1​,W2​,W3​) for feed-forward networks.
• Block & Transformer Architecture: Assembling residual connections, RMSNorm, causal masking, KV caching for inference, and cross-entropy loss adaptation for training.

Who Should Attend?

• AI Engineers, Researchers, and Developers looking to understand modern open-weights LLM optimizations.
• Python/PyTorch practitioners wanting a clear implementation guide for MLA, RoPE scaling, and MoE.