London Text Analytics Reboot!

We are excited to announce the return of London Text Analytics!

Our next program will take place in-person at Signal AI's offices on Wednesday 23 July, 2025. Join us for an in-person evening of talks, discussion, and networking focused on NLP research and applications.

Doors will open 6:00pm, with the first talk beginning at 6:30pm.

Drinks and snacks will be provided. The event is free to attend, but space is limited—please RVSP to secure a place. We’d love to see you there.

Speaker:
Chris Madge

Title:
Reasoning, Ambiguity, and Clarification: Probing LLMs in Interactive Minecraft Tasks

Abstract:
We examine the capabilities and challenges of using Large Language Models (LLMs) in task-oriented dialogue settings, particularly situated dynamic Minecraft-like environments. Our work focuses on two interconnected aspects: using LLMs as Minecraft agents in builder and architect roles, and their ability to ask clarification questions in asynchronous instruction-giver/instruction-follower settings. To achieve this we prepared a new unified corpus that combines annotations for reference, ambiguity, and discourse structure, enabling systematic evaluation of clarification behavior. Through platform-based interaction and comparison with human data, we find notable differences: humans rarely ask clarification questions for referential ambiguity but often do for task uncertainty, while LLMs show the opposite tendency. We further explore whether LLMs’ question-asking behavior is influenced by their reasoning capabilities, observing that explicit reasoning increases both the frequency and relevance of clarification questions. Our findings highlight both the promise and current limitations of LLMs in handling ambiguity and improving interactive task performance.

Lightning Talks: The presentation will be followed by lightning talks, to which we invite all attendees to contribute. We will allocate 30-45 minutes, and talks should be around 5 minutes long. It could be about an idea, a recent project or even a proposal for plans or activities for this meetup. This will be fairly informal and we don't expect slides but feel free to present slides if you would like.

Location: Signal AI, 4th Floor, The Script, 44 Featherstone St, London EC1Y 8RN