AGENDA
06:00 pm - Welcome, Rudolf Grötz
06:05 pm - Adaptive Automation by Visual Recognition - Theodor Hartmann (Product Manager) and Johann-Sebastian Zeisz (Product Engineer)
06:45 pm - Q&A
07:00 pm Call it the Day
+++ Adaptive Automation by Visual Recognition
Why heal tests when you can prevent the pain? Learn how adaptive automation by visual recognition uses lightweight machine-learning models to identify and interact with UI elements based on their appearance and context - resilient against evolving interfaces and brittle selectors. We’ll also show how this philosophy powers our drvless solution.
Abstract
In modern agile and DevOps environments, user interfaces change frequently, often breaking traditional selector-based automated tests. “Self-healing” automation tries to patch these failures by updating locators at runtime - but this reactive approach still depends on fragile underlying identifiers and can introduce hidden risks.
Pure image-based testing has been around for years, but traditional template matching is brittle - breaking with minor styling changes, scaling adjustments, or resolution differences. Our counterpoint to this is adaptive automation by visual recognition, powered by lightweight machine-learning models. These models detect UI elements (buttons, icons, controls) based on learned visual features and surrounding context — not exact pixels and not heavyweight language models.
This approach makes tests inherently more resilient because it identifies elements by learned characteristics - shapes, icons, spatial relationships - rather than relying on brittle selectors or exact pixel matches. By focusing on what the user actually sees, the automation can locate and interact with controls even when their internal IDs, positions, or styles change. Of course, no method is flawless: lightweight ML models may still face challenges with fast-moving animated elements, context-sensitive visibility, or overlapping UI components. Understanding these boundaries is key to applying visual recognition effectively - and in this session, we’ll show practical strategies to work within and around them.
We’ll explore why ML-based visual recognition delivers a fundamentally different and proactive testing strategy, walk through its general approach, and address common pitfalls such as environmental variability, performance trade-offs, and distinguishing similar-looking elements. We’ll then show how objentis’ drvless solution applies these models with precise object detection and context-aware automation.
The result: tests that break less often, require less locator maintenance, and fit seamlessly into modern CI/CD pipelines - a practical alternative to “self-healing” approaches.
