Polyglot Battlemode: Contribute to an Open Evaluation LLM Dataset
Details
How well do today's AI models actually perform across different languages? And what happens when we evaluate them through the eyes of native speakers?
Join us for a hands-on, community-driven session where we'll collaboratively build an open multilingual LLM evaluation dataset.
Using Polyglot Battlemode, we'll explore how different language models perform on real-world tasks, comparing their responses for accuracy, usefulness, reasoning, instruction following, and linguistic naturalness.
What we'll do:
- Create and contribute realistic prompts in different languages.
- For each prompt, compare responses from 2 leading AI models side by side.
- Evaluate what works, what fails, and where models struggle with linguistic and cultural nuances.
- Contribute prompts, human judgments, and insights to an openly available evaluation dataset.
Our goal is to build reusable evaluation resources that reflect how people actually use AI across languages, informed by the expertise and perspectives of the communities that speak them.
Who should join?
Native and fluent speakers of any language (Japanese, Indonesian, Korean, German, French, Spanish, Hindi and many more!), researchers, engineers, linguists, and anyone curious about AI evaluation. No coding or prior evaluation experience is required.
Bring your language, your perspective, and your curiosity. Let's build better multilingual AI evaluations together!
Instructions will be sent 2 days prior to the event. Join us on Discord to chat!
