Pre-Haystack US Meetup

Join us the evening before Haystack for this in-person meetup in Charlottesville! We've got a couple of talks planned by Eric Pugh and Daniel Wrigley from OpenSource Connections.

Obviously, not every member of this group will be in town and we don't plan to stream or record this meetup, but you can still join the main Haystack conference remotely via: https://www.eventbee.com/t/292625631/meetup!

Agenda
18:00 - Doors open
18:30- First talk (Eric Pugh)
19:15 - Networking & pizza
19:45 - Second talk (Daniel Wrigley)
20:30 - Let's socialize ;-)

Talks

Can I use a really Stoopid LLM as a Judge and STILL get some Value? Experiments with Ollama models running on the CPU
Eric Pugh

In this talk Eric will demonstrate how easy it is to play with three models in Quepid, starting with industry standard gpt-4o and then swapping to the 398 MB qwen-2.5:0.5b and llama3.2:1b models. We’ll look at some inter rater reliability statistics and then discuss if simplistic models you run on your laptop bring anything to the table!

Measuring Search Relevance with User Behavior Data in OpenSearch
Daniel Wrigley

Ensuring high-quality search results is a complex and ongoing challenge for search engineers. As data, ranking algorithms, and search platforms evolve, measuring relevance effectively becomes even more critical—yet also increasingly difficult. Traditional evaluation methods often require significant resources, making it particularly challenging for small organizations or teams without dedicated search expertise.

In this talk, we will explore how user behavior data can be leveraged to measure search relevance within OpenSearch. We'll discuss key challenges, such as position bias (users preferring higher-ranked results) and data sparsity, and how these factors impact implicit judgments. We will show how User Behavior Insights (UBI) can be leveraged as a tool for collecting and analyzing behavioral signals, and how the search quality evaluation app can transform these raw interaction signals into meaningful search relevance metrics.
The session will include a live demo showcasing how to collect user behavior data with UBI, derive implicit judgments, and use these insights to quantify search quality over time. Attendees will leave with practical strategies to improve search relevance measurement, even in resource-constrained environments.