Data Labeling for Search Relevance Evaluation

We're excited to welcome Dr. Dmitry Ustalov, Head of Research at Toloka, who will share his experience in building human-in-the-loop pipelines for information retrieval evaluation based on our full-day tutorial presented at the WWW'21 conference. There will be an introduction of the ranking problem, a discussion on the commonly used ranking quality metrics, and then focus on a human-in-the-loop-based approach to obtain relevance judgments at scale. These judgments can be further used to improve the performance of search and recommender systems. Finally, he will share and discuss best practices and pitfalls from his own experience.

📌 If there is anything particular that you'd like to learn from the talk or ask us beforehand please fill out the form: https://toloka.ai/academy/ml-tokyo

👉 Speaker Bio: Dr. Dmitry Ustalov is the Head of Research at Toloka, a global data labeling platform. He is responsible for enabling the state-of-the-art methods for quality control in Toloka and spreading the innovations made by the Toloka Research team. https://www.linkedin.com/in/ustalov/

👉 JOIN ZOOM
https://us02web.zoom.us/j/82607190299?pwd=dlZWVUdKNStTRmtFU2VLQy8xMGFodz09

● MLT NEWSLETTER
Sign up for the (infrequent and low noise) newsletter if you'd like to stay up to date with events and what we're working on https://www.getrevue.co/profile/mltai

● MLT PATRON
Become an MLT Patron and help us to keep MLT meetups like this inclusive and for free. https://www.patreon.com/MLTOKYO

● FIND MLT RESOURCES
Github: https://github.com/Machine-Learning-Tokyo
Youtube: https://www.youtube.com/MLTOKYO
Slack: https://bit.ly/36ImxtW

● RECRUITING
MLT events are for community building and knowledge sharing. We politely ask that company representatives, recruiters, and consultants looking to hire or sell their services do not participate in MLT activities or approach members in any form.

● CODE OF CONDUCT
MLT promotes an inclusive environment that values integrity, openness, and respect. https://github.com/Machine-Learning-Tokyo/MLT_starterkit