From Seasonality-Aware Adaptive Experimentation to Survival Modeling

Booking.com Machine Learning Webinar Series is all about bringing our extensive ML community knowledge to ML practitioners around the world.

We invite Booking.com brightest ML scientists and engineers to share their experience and insights on a wide variety of topics across Booking's ML ecosystem.

Agenda

Seasonality-aware adaptive experimentation
Christina Katsimerou
Principal Machine Learning Scientist, Booking.com

E-commerce firms use online controlled experiments, or A/B/n testing, for evidence based decision making, learning, and inspiration. The most straightforward way to conduct an A/B/n experiment is distribute the traffic uniformly among the default product (control) and the alternative versions (variants); the variants that appear to be significantly better than the control at the (fixed in advance) stopping time of the experiment are considered for deployment. While well-established, this process can be inefficient in terms of resources and prone to human error. At Booking, we are building a new experimentation platform, relying on sequential testing policies that can both adjust the allocation of the samples and be stopped adaptively. The platform needs to be able to learn under data heterogeneities that are present in real world traffic, such as daily or weekly seasonality. In the talk, I will describe the subpopulation-aware mathematical model and provide results from an A/B/n experiment that show the improvements in sample efficiency compared to a learner that assumes sample homogeneity.

Survival Modeling
Jacob Lynn
Senior Machine Learning Scientist, Booking.com

Survival modeling approaches are used in the presence of censored and delayed data, indicating that the event of interest has not (yet) occurred during the study period. These approaches are also known as time-to-event modeling and are often applicable to customer retention and churn.
Within Booking.com, predicting cancellations is useful across many areas of the business. Historically we have done this with traditional classification models, but the dramatic changes in cancellation behavior caused by the pandemic exposed limitations in this approach.
In this talk we describe a new survival modeling based approach and its application to predicting cancellations.