The Testing Gap: Why Classic Automation Falls Short with AI Systems

Hi everyone, time for another PyData Cyprus meetup. Georgio will talk to us about testing and automation and what is different when we deal with AI systems.

This will be an online-only event. However, soon we will have physical events in Cyprus.

Title
The Testing Gap: Why Classic Automation Falls Short with AI Systems

Abstract
As AI-driven systems become increasingly embedded in modern software products, classic automation frameworks struggle to keep pace. In this talk, we’ll explore the limitations of traditional test automation when applied to machine learning components — from unpredictable outputs to non-deterministic behaviour and evaluation metrics beyond pass/fail. We’ll focus on how Playwright, a modern end-to-end testing framework, can be extended to better support AI-infused applications. Key strategies will include human-in-the-loop validation, golden datasets, adaptive assertions, and continuous model monitoring.