A/B Testing for AI
Details
Free online AI tech talk event, you can join from anywhere with zoom, Register and attend:
https://learn.xnextcon.com/event/eventdetails/W2020111610
Abstract:
A.I. systems are complex. They are trained on massive datasets and adapt their behavior on the fly. Their real impact on users is often unclear until the system is deployed in the real world. In this sense, the field of A.I. is similar to medicine, where the impact of a drug is not know until it is tested on real people. Yet, while in medicine FDA demands that every drug undergoes a rigorous evaluation via a randomized clinical trial, most A.I. practitioners employ very weak evaluation procedures that rely on offline metrics such as precision, recall, AUC, etc.
In this talk, I will discuss issues resulting from this lack of rigor, and show how A/B testing can help detect bias, catch performance issues, obtain deeper insights, and ensure better user experience when developing and deploying A.I. systems.
I will share examples, pitfalls and lessons learned from over a decade of my experience of applying A/B testing to evaluate A.I. systems at Outreach and Microsoft. These lessons will help anyone who wants to apply A/B testing to A.I. scenarios do it correctly and maximize their learnings.
Speakers:
Pavel Dmitriev, VP of Data Science at Outreach,
Networking:
30mins before/after the event on slack. Join slack by the invitation: https://bit.ly/3gi7bjf . The two channels:
#jobs for job posting from speakers, partners, sponsors companies, and you can Q&A with hiring managers right in the channel.
#events for events Q&A, mixing and networking with speakers and other peer attendees.
