How Bayesian Statistics Solves Common Problems when Analyzing Large Samples

Robert Neal is Principal Software Engineer on Experimentation and Analytics Platform at Udemy
Graduate Student in Epistemology, Ethics & Mind at University of Edinburgh

His focus recently is on experimentation methodology and data science engineering. Next to that he is working on research in applied behavioral economics. Robert will talk about challenges with experimentation methodologies (e.g. null hypothesis significance testing) and pitfalls in analysis (e.g. multiple comparison problems) that are particularly problematic when experimenting at scale. We have terrabytes of data that we are generating and as a company we often want to adopt variations of experiments that have very small effects. However, classical statistics is not well-suited to situations like this. So we have adopted a Bayesian approach to experimentation analysis that is simpler (i.e. requires less modeling) and provides more information (i.e. information about uncertainty). Robert will dive into the problems they have faced and how they solved them both at his previous employer Credit Karma, and how they are now approaching it at Udemy.