SEA: Search Engines Amsterdam

This meetup session will focus on counterfactual methods for learning-to-rank and evaluation. It will feature two speakers: Olivier Jeunen (independent researcher and Booking.com ML consultant) and Harrie Oosterhuis (Associate Professor at IRLab, University of Amsterdam).

Location: Science Park - LAB42, Room L1.17
Date & Time: Friday, October 30, 16:00–17:00 (CEST)
Zoom link: https://uva-live.zoom.us/j/65011610507

We’ll also post our future events on GatherEasy. Here is the registration link for this event: https://gathereasy.rsvp/event/aEmipmREB

Details below:

Speaker #1: Olivier Jeunen
Title: Accelerating A/B-Tests with Counterfactual Estimation

Abstract:
A/B-tests guide decision-making across online platforms, but the need for statistical power often makes them expensive. In the case of stochastic policies, overlap in action distributions (e.g. two policies recommending similar slates of items) allows us to reduce variance and reach reliable conclusions faster.
In this talk, I show how connecting online A/B-testing with off-policy evaluation lets us focus on the decisions that distinguish competing systems. By accounting for where systems are likely to agree and disagree, we can reduce noise in treatment effect estimates and improve power. I’ll explain the intuition, present theoretical results and simulation evidence, and discuss how policy overlap can guide experimental design, with applications to recommender systems, search result rankings, and beyond.

Bio:
Olivier Jeunen is an independent researcher and machine learning consultant collaborating with Booking.com on advertising and relevance. His research focuses on recommender systems, online advertising and experimentation, with particular emphasis on causal inference and off-policy evaluation. He obtained his PhD in Computer Science from the University of Antwerp in 2021 and previously held research positions at Aampe, ShareChat, Amazon, Spotify, Meta and Criteo.

Speaker #2: Harrie Oosterhuis
Title: From Inverse-Propensity-Scoring to Doubly Robust Estimation: Counterfactual Learning to Rank for Search and Recommendation

Abstract:
Search and recommendation systems are vital for the accessibility of content on the internet. The basis for these systems are ranking models that turn collections of items into rankings: small, ordered lists of items to be displayed to users. Modern ranking models are mostly optimized based on user interactions. Generally, learning from user behavior leads to systems that receive more user engagement than those optimized based on expert judgements. However, user interactions are biased indicators of user preference: often whether something is interacted has less to do with preference and more with where and how it was presented.
In response to this bias problem, recent years have seen the introduction and development of the counterfactual Learning to Rank (LTR) field. This field covers methods that learn from historical user interactions, i.e. click logs, and aim to optimize ranking models w.r.t. the actual user preferences. In order to achieve this goal, counterfactual LTR methods have to correct the biases that affect clicks.In this talk I will present a recently introduced doubly robust method for correcting position-bias in user interactions with rankings.

Bio:
Dr. Harrie Oosterhuis is an Associate Professor at the Informatics Institute of the University of Amsterdam. His research lies on the intersection of Machine Learning, Information Retrieval and Recommendation Systems and primarily concerns statistical methods for optimizing ranking models. Much of his work combines theory and practice; in recent years, Harrie has combined his academic position with part-time research positions at Google DeepMind, Twitter, and currently Amazon Music. He graduated his PhD cum laude at the University of Amsterdam in 2020.