Skip to content

Details

Review programming fundamentals. Pytorch, HuggingFace apis, The popularity of LLMs has caused extensive copy and paste in open source projects such as vllm. We are going to cover some basic programming idioms needed for distilling models from big to smaller models. These skills are transferrable and are different than the classic RL exercises.
pre distillation exercises

You will need a colab account. We will review each problem and leave time for everyone to complete their solutions without the aid of a LLM.
This event is about repetition and practice.

  1. HF apis to tokenizer and model
  2. how to speed up model downloads
  3. basic idioms, batching, padding, attention masks
  4. logits and probability distributions with softmax
  5. minimizing KL divergence
  6. top-k and softening teacher distribution by raising T. Dark Knowledge
  7. token level knowledge distillation. (distillation event we will stop here)
  8. Sequence level distillation
  9. DAgger Ross 2011
  10. Off policy Distillation
  11. On policy Distillation Agarwal 2024
  12. Self Distilled Reasoning Zhao 2016
  13. Self Distillation Fine Tuning Shenfeld 2026

Generated by Codex.
[https://colab.research.google.com/drive/1EokTq_U-_JcPHtPTTjfVUcSjTW7dyDiB?usp=drive_link](lab)
Zoom link 10m before 630pm time on Thu.

You may also like