Silicon Valley Hands On Programming Events

Deep learning is evolving quickly. Important new developments are appearing daily. This group attempts to keep up by reading and discussing current deep learning literature. This meetup uses discussion among the participants to speed understanding of current research results. That requires that some participants read the paper before attending. Anyone is welcome to attend and listen without reading the paper. If nobody reads the paper the meeting will be short.

Paper for August 11, 2026:
Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
https://arxiv.org/abs/2603.10123

Papers that we're reading, code that participants generate and other random stuff can be found at github site for the group.
https://github.com/davidmacmillan/DeepLearningStudyGroup

This is a working group. There is no formal lecture or teaching format. Do your own demos like it is work and build a set of demos.

Model selection and evaluation

• How would you choose between two open models for a customer?
• Why is leaderboard performance insufficient?
• Design an evaluation suite for a customer-support agent.
• How do you evaluate tool-call correctness?
• When should the customer use prompting, RAG, SFT, DPO or RFT?
• How would you detect regression after changing models?
• What should be masked when training tool-calling trajectories?
• How do you prevent benchmark contamination?
• How would you measure quality versus latency and cost?

Take home project

## Exercise 1: Model migration and inference benchmark

Timebox: 4 hours

### Scenario

A customer currently uses a closed-model API for processing customer tickets at web portal. They want to migrate to an open model served through Together AI.
Workload:

• 3 million requests per month
• Median input: 3,000 tokens
• p95 input: 8,000 tokens
• Average output: 300 tokens
• Peak traffic: 10 requests/second
• p95 TTFT target: 800 ms
• Output must follow a JSON schema
• Quality cannot decline by more than 3%

### Assignment

Compare two open models and recommend one.
Implement:

```
POST /summarize
```

Response:

```
{
"summary": "string",
"key_points": ["string"],
"risk_level": "low | medium | high"
}
```

Measure:

• JSON validity
• Summary completeness
• Factual consistency
• TTFT
• End-to-end latency
• Input/output throughput
• Estimated cost per 1,000 requests

### Deliverables

```
app.py
benchmark.py
evaluation.py
test_cases.jsonl
results.json
README.md
```

### Follow-up questions

• Why did you select these models?
• How would batching affect TTFT?
• When would a dedicated endpoint become economical?
• How would you handle 10× traffic?
• Why might the fastest model produce the worst business result?

Deep learning is evolving quickly. Important new developments are appearing daily. This group attempts to keep up by reading and discussing current deep learning literature. This meetup uses discussion among the participants to speed understanding of current research results. That requires that some participants read the paper before attending. Anyone is welcome to attend and listen without reading the paper. If nobody reads the paper the meeting will be short.

TBD

Papers that we're reading, code that participants generate and other random stuff can be found at github site for the group.
https://github.com/davidmacmillan/DeepLearningStudyGroup

Deep learning is evolving quickly. Important new developments are appearing daily. This group attempts to keep up by reading and discussing current deep learning literature. This meetup uses discussion among the participants to speed understanding of current research results. That requires that some participants read the paper before attending. Anyone is welcome to attend and listen without reading the paper. If nobody reads the paper the meeting will be short.

TBD

Papers that we're reading, code that participants generate and other random stuff can be found at github site for the group.
https://github.com/davidmacmillan/DeepLearningStudyGroup