Multi-Variate (A/B) Testing at Expedia

Multi-Variate (A/B) Testing at Expedia

By Ralph Jones, Engineering Manager, Experiment Engineering

Expedia has a strong Test and Learn ethos, driven by a desire to truly understand and delight our customers. We believe that it is better to try new features and learn fast – with success or failure - than not to try at all. It’s also important to know how big a success (or failure!) that new feature was.

We have a proven A/B testing framework and run hundreds of experiments across all elements of our technology stack. Our expert Experiment Engineering team provide an in-house toolkit to the rest of the organisation that enable them to implement, manage and run their tests. The results of those tests are delivered in near real-time, providing a way to quickly validate and course-correct the new features being tested.

Behind all of this is a system that deals with over 70,000 experimentation requests per second, including a big data pipeline which analyses terabytes of data.

Over beers and pizza Ralph will introduce the approach Expedia has chosen, demo our in-house tools, give an overview of the technologies in play and - perhaps most importantly - the hard lessons we've learned along the way.