MEL MLOps Community #19 | Advanced Evals & Deep Agent Harnesses
Details
We are stoked to announce our next Melbourne MLOps Community Meetup of the year!
We’re gathering at Mantel to unpack the realities of building reliable, production-ready AI agents. Join two top-tier industry experts to learn how to move beyond basic evaluations and build the structural harnesses that turn fragile demos into real-world tools.
🎤 Our speakers and topics:
🧪🎯 The Lesser of Two Evals: Why Your Agent Passes Benchmarks and Fails Users
Josh Gillies, Senior Software Engineer @ Prefactor
Traditional LLM evaluation metrics (correctness, groundedness, and hallucination checks) are necessary starting points, but they aren't enough to guarantee a great agent. In production, agents encounter unexpected multi-turn branching, bad conversations, and tools which can fail in a myriad of ways. In this talk, we’ll explore evaluations beyond using a single-prompt LLM as a judge and look at Agent quality as a holistic system.
🏗️🧠 Your Agent Needs a Harness: How deep agents actually work
Lisa Sherin, Founding Deployed Engineer @ Langchain
Most agent demos work because someone is watching. The gap between a demo and something you'd let run unattended isn't the model, it's the harness around it: how you structure the loop, hand it tools, manage context, and recover when a step goes wrong. This session pulls apart the agent harness pattern and how Deep Agents use it to plan, delegate, and stay on task over long-running work.
Networking
The Melbourne MLOps Community is committed to fostering a space where AI/ML engineers and practitioners can connect, exchange ideas, and build meaningful relationships.
Come ready to discuss:
- The tools you’re experimenting with
- Trends you’re seeing in ML and MLOps
- Challenges you’re tackling
- Ways to find peer support
We’ll have food, drinks, and the perfect space for great conversations — so feel free to bring along anyone interested in ML, MLOps, or AI more broadly.
Agenda
🍕 5:30pm – Networking, drinks & food
🙋 6:00pm – Welcome
🎤 6:10pm – Presentation by Josh
❓ 6:30pm – Questions for Josh
🎤 6:40pm – Presentation by Lisa
❓ 7:00pm – Questions for Lisa
🍻 7:10pm onwards – Networking, drinks & food
We’re looking forward to seeing you there!
P.S. If you can’t make it in person, we’ll be streaming the event via Google Meet.
Google Meet link: https://meet.google.com/fxh-hpsg-ryt