[UCL WI Talks]: User Simulation in Mixed-Initiative Conversational Search

Abstract:
While conversational search systems enjoy profuse advancements across multiple aspects, their evaluation remains arduous. The challenge arises from the fact that proper evaluation usually requires expensive and time-consuming user studies, while offline methodologies rarely mimic real-world scenarios. Recently, user simulation has been proposed to tackle these shortcomings.

In this talk, I will present our work on LLM-based user simulators, capable of assuming a user’s role in multi-turn interactions with conversational search systems. To this end, we design simulators to answer potential clarifying questions posed by the system, as well as to provide explicit feedback to system’s responses. Moreover, we showcase the benefit of simulating users, by developing methods for effective feedback utilization, leading to significant improvements in system’s retrieval performance. Finally, we touch on user simulation for task-oriented dialogue systems and discuss the advantages, as well as limitations, of LLM-based simulators.

Bio:
Ivan Sekulić is a final year PhD student at USI Lugano, Switzerland, supervised by Prof. Fabio Crestani. His research interest lie in information retrieval and natural language processing, with a focus on clarification and user simulation in mixed-initiative conversational search. Over the last few years, he worked on multiple IR and NLP topics during research visits to several renowned institutions, including Heidelberg, Singapore, Glasgow, Stavanger, and Zürich. His collaborations resulted in published work at top-tier venues, including SIGIR, ECIR, ACL, and earned him best paper runner-up award at WSDM’22.