Speculative Decoding: Deconstructed — A 3-Hour Hands-On Lab

Speculative Decoding: Deconstructed — A 3-Hour Hands-On Lab (Bring Your Laptop)

Two models. One consumer GPU. Zero quality loss. Up to 2–3× faster LLM inference — and you'll build it from scratch before you ever touch a library flag.

Speculative decoding is the optimization running inside every serious inference stack today — vLLM, TensorRT-LLM, llama.cpp — yet most engineers treat it as a magic config flag. In this session we deconstruct the abstraction: you'll measure why autoregressive decoding is memory-bound, write a complete speculative decoder in ~60 lines of PyTorch, prove its losslessness empirically, and then run the production implementations to see the exact same physics at scale.

No HPC. No cloud credits. The entire workshop is engineered to fit in 6 GB of VRAM — a Qwen2.5-3B target (4-bit) paired with a Qwen2.5-0.5B draft. If it runs on the laptop in your backpack, you'll actually run it again tomorrow.

What you'll walk away having done (not watched):

• Measured, with a stopwatch, that verifying 8 tokens costs the same as generating 1 — the entire economic case for speculative decoding
• Built a working speculative decoder from first principles, including the infamous verify-slice off-by-one
• Implemented the rejection-sampling rule from the original Google/DeepMind papers and proven — by histogram, not by algebra — that the output distribution is untouched
• Benchmarked acceptance rates across code, factual, and creative prompts, and learned when speculative decoding becomes a slowdown
• Run production speculative decoding two ways: Hugging Face assisted generation (one line!) and llama.cpp's `llama-speculative`, including deploying it as an OpenAI-compatible server

## Agenda (3 hours, minute-by-minute)

| Time | Module | Format | Outcome |
| ---- | ------ | ------ | ------- |
| 0:00–0:10 | Welcome + VRAM budget walkthrough | Talk | Everyone knows exactly what fits where in 6 GB |
| 0:10–0:35 | Module 1 — Why decoding is slow (memory-bound math) | Talk + Lab 1 | Attendees measure that verifying 8 tokens costs the same as generating 1 |
| 0:35–0:50 | Module 2 — The speculative decoding algorithm (whiteboard) | Talk | Draft-verify-accept loop + acceptance math internalized |
| 0:50–1:35 | Module 3 — Build it from scratch (Lab 2: greedy) | Lab | Working speculative decoder in ~60 lines of PyTorch |
| 1:35–1:45 | Break | — | — |
| 1:45–2:10 | Module 4 — Losslessness: rejection sampling (Lab 3) | Talk + Lab | Sampling version + proof-by-experiment that output distribution is unchanged |
| 2:10–2:30 | Module 5 — Production path #1: HF Transformers assisted generation (Lab 4) | Lab | One-line speedup + acceptance-rate benchmarking |
| 2:30–2:50 | Module 6 — Production path #2: llama.cpp `llama-speculative` (Lab 5) | Lab | GGUF-based speculative decoding + prompt-lookup bonus |
| 2:50–3:00 | Benchmark showdown + wrap-up + quiz | Discussion | Numbers on the board; when specdec wins and when it doesn't |

Who this is for: Backend/platform/ML engineers, SREs, and students who are comfortable with Python and want to understand LLM inference at the kernel-truth level. No prior inference-optimization experience needed.

Bring: A Linux/WSL2 laptop with an NVIDIA GPU (≥6 GB VRAM) and ~12 GB free disk. No GPU? Still come — there's a full CPU-only track via llama.cpp, and every concept transfers 1:1

Before you arrive: A pre-workshop setup guide (Python env + model downloads, ~15 min) will be shared with registered attendees — please complete it so we can spend all 3 hours building, not downloading.
Seats are limited to keep it truly hands-on. Register early. ☕