Skip to content

Details

We have had consistent zoom bombing problems. Link up 10m before start time

vllm ticket review: https://github.com/vllm-project/vllm/issues/50128

Interview Question
Take home project from interview

## Model migration and inference benchmark

4 hours

### Scenario

A customer currently uses a closed-model API for processing customer tickets at web portal. They want to migrate to an open model served through a cloud GPU provider
Workload:

  • 3 million requests per month
  • Median input: 3,000 tokens
  • p95 input: 8,000 tokens
  • Average output: 300 tokens
  • Peak traffic: 10 requests/second
  • p95 TTFT target: 800 ms
  • Output must follow a JSON schema
  • Quality cannot decline by more than 3%

### Assignment

Compare two open models and recommend one.
Implement:

```
POST /summarize
```

Response:

```
{
"summary": "string",
"key_points": ["string"],
"risk_level": "low | medium | high"
}
```

Measure:

  • JSON validity
  • Summary completeness
  • Factual consistency
  • TTFT
  • End-to-end latency
  • Input/output throughput
  • Estimated cost per 1,000 requests

### Deliverables

```
app.py
benchmark.py
evaluation.py
test_cases.jsonl
results.json
README.md
```

Model selection and evaluation

  • How would you choose between two open models for a customer?
  • Why is leaderboard performance insufficient?
  • Design an evaluation suite for a customer-support agent.
  • How do you evaluate tool-call correctness?
  • When should the customer use prompting, RAG, SFT, DPO or RFT?
  • How would you detect regression after changing models?
  • What should be masked when training tool-calling trajectories?
  • How do you prevent benchmark contamination?
  • How would you measure quality versus latency and cost?

### Follow-up questions

  • Why did you select these models?
  • How would batching affect TTFT?
  • When would a dedicated endpoint become economical?
  • How would you handle 10× traffic?
  • Why might the fastest model produce the worst business result?

Example: running llm.d with 2 GPUs and reporting some numbers doesn't do you any good, comparing to another model with less or more weights and seeing the performacnce/cost tradeoff is more meaningful. Your choice on which configs to report will help you get jobs.

You may also like