cs329z Agentic AI Working Group-- RAG programming session

There are thousands of agentic programming jobs. Search in linkedin.

We are going to split you up into groups. Doing demos doesn't build portofilios

Examples of Agentic Systems:
https://taubench.com/
Taubench2 uses RAG, the optional args on the CLI runs the set of tasks. Eval the RAG archicture features not by visual feel but by the task error rate. The less error rate the better your RAG system, assuming the error is caused by RAG.

https://x.com/dotey/status/2120388990616433083

Stanford CS329Z course materials now public: lectures go up on YouTube daily, two assignments open-sourced, the first teaches you to build a “Claude Code” from scratch for a few dollars Stanford’s new course CS329Z, “Engineering AI Agents,” has begun making all its course materials publicly available. In a post on November 10, one of the instructors, Michael Ryan, said lecture videos are being released one at a time each day on Stanford Online’s YouTube channel, and the code repositories for the two assignments have also been posted on GitHub, both under the MIT license, so other schools can use them for teaching as long as they credit the source. When the course launched, things were different. The course page said the recordings would be available only on Stanford’s internal Canvas platform for enrolled students, and people outside the university could only download the course materials. The official channel has now posted Lecture 1, “What Is an Agent System,” and Lecture 2, “Large Language Model Knowledge for Agent Developers,” and Lecture 1 has passed 90,000 views. The course homepage schedule has also started linking a video next to each class. 【Assignment 1: Build an agent harness yourself】 A harness is the layer of software wrapped around a model: it tells the model what tools it has, parses its replies, runs commands on its behalf, and manages context. Claude Code and Codex can both be seen as a model plus a harness. The assignment forbids any agent framework. The course provides only the most primitive interface: pass in a list of messages and return a block of text, with no tool calling and no structured output. Everything else is up to you to write. The first half has you build a few fixed workflows: prioritize emails, generate a daily digest, hand-write BM25 retrieval (a classic search algorithm that scores by keywords), and then do multi-hop question answering over a real company’s email archive. The second half has you build an internal assistant for a fictional energy company: design your own tool-calling format, connect document retrieval, give it a terminal restricted by a sandbox, compress the context when a conversation gets too long, remember things about the user across sessions, and have the user approve or reject actions before they are carried out. Finally, a large language model playing a “simulated user” holds a multi-turn conversation with your agent, and another model scores it. For self-learners, the most useful part is that the full test suite is in the repository. At the start, every test fails, and your job is to make them pass one by one. Of the 88 points in the automatic grading, 48 come from tests that never call a model, so you can run them as many times as you want without spending anything. There are also a few judgment questions for which the assignment gives no standard answer—for example, how much of an overly long command output to show the model, what must be kept when compressing context, and which operations can be done without asking the user—and it asks you to write a design document explaining your choices. Michael Ryan said an API budget of under $5 is enough to finish it. The numbers in the handout are lower: running the full online test suite once with the reference solution cost $0.05, and the final simulated-user evaluation cost another $0.18. Debugging on your own means running it over and over, so it will cost more than that, but every call is cached locally, and the same request is not charged a second time. Enrolled students use the OpenRouter key issued by the course; people outside the university have to get their own, and the default model is openai/gpt-6-luna. 【Assignment 2: Write an exam question for an agent】 The second assignment was designed by John Yang. He is a co-author of SWE-bench, a benchmark that uses real GitHub issues to test whether an AI can fix bugs, and companies often cite it when reporting results for their coding models. The assignment teaches how to write a Terminal-Bench-style question: give an agent a problem description, have it work in a command-line environment, and then use a test script to judge whether it did the job correctly. The tool it uses, Harbor, comes from the Terminal-Bench team. It defines how a question is packaged (the problem statement, the Docker environment, and the verification script), and it handles launching the agent, running verification, and saving the results. The assignment first walks you through using Harbor to run existing questions from SWE-bench Verified and Terminal-Bench 2.1. The part worth 70% of the grade is writing a question of your own, running it five times with mini-SWE-agent (a minimal coding agent that has only a bash tool) paired with Gemini 3.8 Flash, and analyzing whether the question is valid, close to real-world conditions, reproducible, and difficult. You need to install Docker and have a Gemini or OpenRouter key ready. Both assignment write-ups end with an acknowledgment: the first was made with extensive help from Claude Fable 5.1, and the second used Claude Opus 5.5. 【What comes later】 The course is taught jointly by Diyi Yang of Stanford’s computer science department and two PhD students, Michael Ryan and John Yang, and the first cohort has more than 100 students. This semester’s schedule also includes a fireside chat with Andrew Ng, and guest lectures by two of the SWE-bench authors, Carlos Jimenez and Alexander Wettig (now at Anthropic). Michael Ryan said student projects will be made public later as well. The main funder is the Laude Institute, and Thinking Machines, Google Cloud, AWS, and Harbor have provided API credits for students. When cloning the repository following the second assignment’s handout, note that the repository name in the handout is assignment2-harness, but the actual address is assignment2-evals. Course homepage: [https://cs329z.stanford.edu](https://t.co/y7KbVMdU2r) Lecture 1 video: [https://youtu.be/DGI4vucR71w](https://t.co/Ls8MuXXJNH) Lecture 2 video: [https://youtu.be/Vr1NzH8bT44](https://t.co/gs46i1u5FX) Assignment 1 repository: [https://github.com/cs329z/assignment1-harness](https://t.co/BzYDnPsmsN) Assignment 2 repository: [https://github.com/cs329z/assignmen](https://t.co/dbAzZW2Zkt)

Videos:
cs329z
Lecture 1: https://www.youtube.com/watch?v=DGI4vucR71w
Lecture 2: https://www.youtube.com/watch?v=Vr1NzH8bT44

https://x.com/kunchenguid/status/2120408893859979511