Skip to content

Details

This is a working group. There is no formal lecture or teaching format. Do your own demos like it is work and build a set of demos.

Model selection and evaluation

  • How would you choose between two open models for a customer?
  • Why is leaderboard performance insufficient?
  • Design an evaluation suite for a customer-support agent.
  • How do you evaluate tool-call correctness?
  • When should the customer use prompting, RAG, SFT, DPO or RFT?
  • How would you detect regression after changing models?
  • What should be masked when training tool-calling trajectories?
  • How do you prevent benchmark contamination?
  • How would you measure quality versus latency and cost?

Take home project

## Exercise 1: Model migration and inference benchmark

Timebox: 4 hours

### Scenario

A customer currently uses a closed-model API for processing customer tickets at web portal. They want to migrate to an open model served through Together AI.
Workload:

  • 3 million requests per month
  • Median input: 3,000 tokens
  • p95 input: 8,000 tokens
  • Average output: 300 tokens
  • Peak traffic: 10 requests/second
  • p95 TTFT target: 800 ms
  • Output must follow a JSON schema
  • Quality cannot decline by more than 3%

### Assignment

Compare two open models and recommend one.
Implement:

```
POST /summarize
```

Response:

```
{
"summary": "string",
"key_points": ["string"],
"risk_level": "low | medium | high"
}
```

Measure:

  • JSON validity
  • Summary completeness
  • Factual consistency
  • TTFT
  • End-to-end latency
  • Input/output throughput
  • Estimated cost per 1,000 requests

### Deliverables

```
app.py
benchmark.py
evaluation.py
test_cases.jsonl
results.json
README.md
```

### Follow-up questions

  • Why did you select these models?
  • How would batching affect TTFT?
  • When would a dedicated endpoint become economical?
  • How would you handle 10× traffic?
  • Why might the fastest model produce the worst business result?

You may also like