From Sequential AB Testing to Fairness in ML

Booking.com Machine Learning Webinar Series is all about bringing our extensive ML community knowledge to ML practitioners around the world.
We invite Booking.com brightest ML scientists and engineers to share their experience and insights on a wide variety of topics across Booking's ML ecosystem.

Agenda

Faster and better experiments with Sequential Testing
Nils Skotara
Principal Data Science & Data Analytics, Booking.com

Experiments allow us to assess the impact our product changes have on the customer.
They are a safety net that detects critical degradations and decides which products improve our website.
A major downside of the way we are currently running experiments is that product teams want to iterate on their experiments as fast as possible to develop promising ideas further and abandon others quickly. As of now, experiments do not accommodate for the need of online experimenters to continuously monitor results. Current testing requires sticking to a fixed runtime. It is nonetheless tempting to act upon the displayed results prematurely, thereby violating the protocol and leading to wrong product decisions.
Sequential testing helps in two ways. It reduces the runtime of experiments by a considerable amount allowing for faster product improvement iterations. It enables experimenters to monitor and immediately act upon results thereby eliminating the danger of increasing the number of wrong decisions.
Further, it frees our users from the need of doing power calculations
Sequential testing is compatible with any kind of statistical test, covariate adjustment, and Non-Inferiority testing. It provides impact estimates as well as confidence intervals.

Fairness in ML: From Theory to Practice
Nadia Tomova
Senior Data Scientist, Booking.com

From college admissions and recommendations to housing loans and risk assessment tools, businesses are relying increasingly on ML for their decision-making. The glaring problem that came to the forefront in the past couple of years is that not all of us are impacted equally and fairly by algorithmic decisions. Concerns around unfair ML have not gone unnoticed by authorities like the EU, who are pushing ML-specific anti-discrimination policies, and it is only a matter of time until audits follow. Most major tech companies like Google, Facebook, Microsoft and IBM are already looking into the dilemma presented by fairness in ML. Some of them even developed tooling to detect, measure and mitigate unfair bias in their models. These issues are also on the [Booking.com](http://booking.com/) machine learning agenda. In 2021, the Responsible AI task force developed a bias detection and mitigation methodology and created our very own in-house tool - B.Fair. Moreover, we ran our first fairness experiment that showed not only improvement in fairness indicators but also significantly improving some key business metrics. In this talk we will discuss the theory behind Fair ML - what is “unfair” vs “fair” ML? What are the benefits of “fair” models? What are the trade-offs? And we go one step further, shifting focus from theory to practice, and discuss our experience running a fairness experiment and our in-house B.Fair tool.