DIR 2021 ½ Week 1: Keynote talk on measuring reproducibility of IR experiments

DIR 2021 ½ has teamed up with SEA and we are proud to be organizing DIR 2021 ½ on four consecutive Fridays.

We are very pleased to announce the first keynote speaker for DIR 2021 ½: Maria Maistro (Uni. of Copenhagen).

Title: How can we measure reproducibility of IR experiments?

Abstract: We are today facing the so-called reproducibility crisis across all areas of science, where researchers have trouble reproducing and confirming previous experimental findings. In Information Retrieval (IR), the most common attitude for reproducibility is some sort of “close enough”: researchers put any reasonable effort to understand how an approach was implemented and how an experiment was conducted. After some iterations, when they obtain performance scores which somehow resemble the original ones, they decide that an experimental result is reproduced. This talk focuses on reproducibility in lR: it presents challenges related to reproducibility, covers some initiatives that have been proposed to ease reproducibility, investigates the problem of “objectively” measuring reproducibility, with reproducibility measures as Kendall’s tau and RMSE to name some, and explores the behaviour and properties of some reproducibility measures.

Short bio: Maria Maistro studied initially Mathematics (BSc, University of Padua, 2011; MSc, University of Padua, 2014) and then Computer Science (PhD, University of Padua, 2018). She is a Marie Curie Fellow and a tenure track assistant professor at the Department of Computer Science, University of Copenhagen (DIKU). She conducts research in information retrieval, and particularly on evaluation, reproducibility and replicability, click log analysis, learning to rank and applied machine learning. She has already co-organized several international scientific events and she has served as member of programme committees and reviewer for highly ranked conferences and journals in information retrieval.

---
Anyone counting? This is talk #189 hosted by SEA.