Split Testing: Everything you wanted to know but were afraid to ask

Abstract:

Split, A/B or multivariate testing are used in a wide variety of contexts and fields as a method to conduct controlled, randomized experiments. In the tech sector these experiments are most often used with the goal of improving a website or app, for example increasing the clickthrough rate on a landing page. What appears at first glance as a simple subject is actually a landscape rich with subtlety and fraught with potential pitfalls that could lead to companies making long term, incorrect decisions.

In this talk I will give a general overview of split testing and four testing methodologies: null hypothesis testing, sequential probability ratio testing, multi-armed bandits, and Bayesian sequential test design. The focus will be high level on the methodologies and I will discuss the pros, cons, and potential pitfalls of each method. This talk will also serve as a survey introduction for a series of more detailed lectures on some of these methods later this summer, but will be self-contained.

About Thomas Levi:

Thomas Levi (https://twitter.com/tslevi) started out with a doctorate in Theoretical Physics and String Theory from the University of Pennsylvania in 2006. His post-doctoral studies in cosmology and string theory, where he wrote 19 papers garnering 720+ citations, then took him to NYU and finally UBC. In 2012, he decided to move into industry, and took on the role of Senior Data Scientist at POF. Thomas has been involved in diverse projects such as behaviour analysis, social network analysis, scam detection, Bot detection, matching algorithms, topic modelling and semantic analysis

Schedule:

• 6:00PM Doors are open, feel free to mingle
• 6:30 Presentation start
• ~7:45 Off to a nearby restaurant for food, drinks, and breakout discussions

Getting There:

By transit there a number of high frequency buses (check Google Maps or the Translink site for your particular case) that will get you there. For the drivers, there is a fair bit of street parking (free and pay) in the area, especially after 6.