data.MAY <- A/B test design: from requirements gathering to conclusion

Talk Description: In this talk, I will shed light on the full cycle of A/B testing. We’ll cover requirements gathering from stakeholders, hypothesis formation, technical preparation, metrics and statistical criterion selection, MDE estimation, multiple comparison correction, SRM detection during user splitting, the peeking problem, and some options of process automation.

We’ll also discuss the bootstrap and delta methods for handling ratio metrics, as well as a variance reduction technique called CUPED.

Speaker Bio: Hi, my name is Roman Feofanov, and I have been working as a Data Analyst for the past five years. Throughout my career, I have conducted numerous A/B tests across a banking product and a streaming service. I failed many of them — and learned even more from those failures. I would be excited to share some real-world cases and insights with you.

🏢 Venue: NewyTechPeople Offices L2, 9 Downie St, Maryville NSW 2293

https://maps.app.goo.gl/QssJqKTbwyaKQMxu9

• We will meet on the Level 2 Seminar Space.

##-------------------------------------------------------------------##

🕠 TIME: 5.30pm - 7.30pm

FOOD: 🍕 + 🥤

• Wood fired pizza, sponsored by NewyTechPeople
• Soft drinks

SCHEDULE:

• 5:30pm: Networking + Food + Drinks

(Don't stress if you are held up in traffic)

• 6:15pm: Presentation
• 7:15pm: Packing up + Bonus questions
• 7:30pm: Close
• 7:30pm+: [Unofficially] anyone wanting to continue chatting, we usually head to a bar (The Maryville Tavern) which is a short walk from the venue.