Skip to content

Details

This is a working group. There is no formal lecture or teaching format. Do your own demos like it is work and build a set of demos.

The company you work for blew their token budget on OpenAI. Your job is to replace the internal demand for llms with open source models running on gpu rentals from a cloud vendor of your choice. Present a POC or usable system consisting of a dashboard, a query router and cost caps per user.

LLM Deployment and Kubernetes:
https://github.com/kserve/kserve
https://github.com/vllm-project/llm-compressor
https://github.com/llm-d/llm-d
https://github.com/GeeeekExplorer/nano-vllm

https://github.com/kvcache-ai/Mooncake

Review Questions

LLM inference

  • Explain the difference between latency, throughput, TTFT and inter-token latency.
  • Why does increasing batch size improve throughput but sometimes hurt latency?
  • What is stored in the KV cache, and why does it often become the memory bottleneck?
  • What is continuous batching?
  • What is paged attention?
  • How does prefix caching help agentic or RAG workloads?
  • What causes GPU underutilization during inference?
  • Compare tensor parallelism, pipeline parallelism and data parallelism.
  • When would speculative decoding help? When might it hurt?
  • How do quantization and reduced precision affect memory, speed and quality?
  • Why can a serving engine outperform vanilla Hugging Face generation?
  • Compare vLLM, SGLang and TensorRT-LLM.

Be prepared to reason about workloads rather than only define terms

platform questions

  • When should a customer use serverless versus a dedicated deployment?
  • How would you migrate an OpenAI-compatible application to Fireworks?
  • How would you deploy a custom Hugging Face model?
  • Why does a LoRA require a dedicated deployment?
  • When would multi-LoRA serving be valuable?
  • How would you configure autoscaling for bursty traffic?
  • How would you implement JSON-schema-constrained output or tool calling?
  • What metrics would you expose to the customer?
  • How would you estimate monthly serving cost?

Know the distinctions:

  • Serverless: shared infrastructure, pay per token, easiest start.
  • Dedicated: private configurable deployment, GPU-second billing, greater control.
  • LoRA: deployed with its base model on dedicated infrastructure.
  • Control plane: manages models and deployments.
  • Data plane: handles inference traffic.

Model selection and evaluation

  • How would you choose between two open models for a customer?
  • Why is leaderboard performance insufficient?
  • Design an evaluation suite for a customer-support agent.
  • How do you evaluate tool-call correctness?
  • When should the customer use prompting, RAG, SFT, DPO or RFT?
  • How would you detect regression after changing models?
  • What should be masked when training tool-calling trajectories?
  • How do you prevent benchmark contamination?
  • How would you measure quality versus latency and cost?

Take home project

## Exercise 1: Model migration and inference benchmark

Timebox: 4 hours

### Scenario

A customer currently uses a closed-model API for document summarization. They want to migrate to an open model served through Together AI.
Workload:

  • 3 million requests per month
  • Median input: 3,000 tokens
  • p95 input: 8,000 tokens
  • Average output: 300 tokens
  • Peak traffic: 10 requests/second
  • p95 TTFT target: 800 ms
  • Output must follow a JSON schema
  • Quality cannot decline by more than 3%

### Assignment

Compare two open models and recommend one.
Implement:

```
POST /summarize
```

Response:

```
{
"summary": "string",
"key_points": ["string"],
"risk_level": "low | medium | high"
}
```

Measure:

  • JSON validity
  • Summary completeness
  • Factual consistency
  • TTFT
  • End-to-end latency
  • Input/output throughput
  • Estimated cost per 1,000 requests

### Deliverables

```
app.py
benchmark.py
evaluation.py
test_cases.jsonl
results.json
README.md
```

### Follow-up questions

  • Why did you select these models?
  • How would batching affect TTFT?
  • When would a dedicated endpoint become economical?
  • How would you handle 10× traffic?
  • Why might the fastest model produce the worst business result?

Example: running llm.d with 2 GPUs and reporting some numbers doesn't do you any good, comparing to another model with less or more weights and seeing the performacnce/cost tradeoff is more meaningful. Your choice on which configs to report will help you get jobs.

You may also like