Techniques for more realistic model evaluations
Details
This week, Axel Ahlqvist will share results from his recent research paper on LLM audit realism, completed at Meridian Cambridge.
As frontier models become ever more capable, they also become better at detecting when they're in a test environment. At that point, they might choose to behave in a way they otherwise wouldn't, so the results of the test can't be trusted. Therefore, a crucial aspect to building a solid understanding of what the latest models are capable of is creating environments that models can't distinguish from real deployment. Axel will present the methods they found to significantly increase test realism, followed by a group discussion.
Paper preprint: Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
📅 September 30th, 18:00
📍 Sveavägen 76 (EA Sweden Office)
🍌 Snacks will be provided
*Optional Reading*
- Automated auditing (Petri): alignment.anthropic.com/2025/petri/
- Paper X thread: x.com/_axelahlqvist/status/2095518876268438009
- Less wrong blog post: lesswrong.com/posts/9DyiNexLoyJqwNWFn/improving-audit-realism-with-inference-time-compute-and
- Paper preprint: arxiv.org/abs/2609.02302
- On eval awareness: anthropic.com/engineering/eval-awareness-browsecomp
We look forward to seeing you there!
(Note that we also post our events on Facebook, so the Meetup attendee list is not indicative of the total expected number of participants.)
Ring "Effektiv Altruism" when you arrive and we'll let you in, then we're up two flights of stairs.