Real world experimentation at Twitter

Using a growth mindset to learn from experience

Please join us for a set of 20 minute talks diving into the cultural, engineering, and analysis aspects of A/B Testing at Twitter!

• 6:00-7:00pm: Networking, food, drinks, and A/B testing product demos and booths from Netflix, Intuit, Uber, Airbnb, and Twitter

• 7:00-8:00pm: Real world experimentation: Using a growth mindset to learn from experience

8:00-9:00pm: More networking, food, drinks, and product demos

Building a culture of experimentation at Twitter

Andy Schlaikjer & Aliza Rosen

Experimentation is critical for data driven product decision making at Twitter. This talk will dive into a case study that explains the how’s and why’s of experimentation at Twitter.

Events to insights: an intro to Twitter’s experimentation system

Nodira Khoussainova & Achal Soni

The experimentation pipeline at Twitter is responsible for computing thousands of metrics for hundreds of experiments from billions of events per day. Learn about our data pipeline architecture, how we’ve improved it over time, and the challenges we still face today.

10 Ways to not mess up your experiments

Evan Reiser

There is a lot of literature and advice about how to run a simple and clean experiment. However, in practice, experimentation is rarely simple and clean. What do you do when things get complex? The greater Twitter team has decades of experience and we’ve made every mistake in the book. We’ll review some of the mistakes you may encounter, and share our pro-tips about how to avoid these pitfalls.

Conversations with experimenters

Theodore Summe

In a panel with A/B testing experts from Twitter, Netflix, Uber, Intuit, and others, Ted will discuss how to run effective experiments that produce actionable output and provide valuable insights into product development.

Bios:

Andy Schlaikjer is an engineer and scientist at Twitter working on the home timeline, ranking, and content recommendation. He also organizes an internal Experiment Shepherds program which connects experts in product experimentation with those running new experiments, and promotes best practices. Prior to joining Twitter in 2011, Andy was a phd student at Carnegie Mellon University, investigating problems in information retrieval and question answering.

Aliza Rosen is a product manager at Twitter working on content creation and engagement. Her work focuses on making it easier for users to express themselves on Twitter. In addition, she works on the structured presentation of Tweets themselves and all of their components across Twitter’s owned and operated products. Prior to Twitter, Aliza worked at LinkedIn and received her bachelor’s degree in Science, Technology, and Society from Stanford.

Nodira Khoussainova is a staff software engineer and tech lead of the Product Insights & Experimentation (PIE) team at Twitter. On the team, she has been involved in building out a company-wide metrics dashboard, a complete overhaul of the experiment results data pipeline, and more. She also consults with product teams to design experiments, define new metrics, and to understand their experiment results. Prior to Twitter, Nodira completed her PhD in Computer Science at the University of Washington, centered around large-scale data analytics.

Achal is a software engineer specializing in big data systems and analytics. Graduating from UC Berkeley from the EECS department with a minor in Applied Mathematics, he joined Twitter’s experimentation team in 2014. He has made key contributions across the board, from the large-scale metric computation pipeline to the internal web dashboard for maintaining and analyzing experiments.

Evan Reiser is the product lead for Twitter’s Core Ads team and is responsible for ads quality, ad serving and machine learning. Previously, Evan founded, built and sold two start-ups focused on A/B testing and experimentation, and was the first product manager at TellApart (an applied machine learning company) which was acquired by Twitter in 2015.

Theodore is a senior product manager building data tools and services. At Twitter, his focus has been on building and refining metrics/experimentation systems as well as ML based personalization for users. Prior to Twitter, Theodore founded the social data company, Discoverly, and previously worked at Salesforce and MorganStanley.

Getting here and other policies:

Public Transportation Options:

BART or MUNI: Both the Civic Center and Van Ness stations are 5 - 10 minute walks from the office. Several bus lines stop right outside of our building, including 6, 7, 21, F, etc.

Caltrain: From the 4th and King station, the MUNI 83X bus stops directly in front of our office and runs every 20 - 25 mins.

Bike Storage:

There is no private bike storage at our office. Your guests may lock on 10th Street at their own risk.

Parking Options:

Other than street parking, we suggest directing your guests to the Civic Center parking garage which is a few blocks down from our office. The entrance is on McAllister Street, between Polk St. and Larkin St, and is open until 12:00 AM.

Press Policy

Twitter reserves the right to refuse entry to this event. Twitter also reserves the right to remove individuals from our premises for disruptive or disorderly behavior, or for any other reason in Twitter's sole discretion.

Live Animal Policy

Twitter HQ has a non-negotiable policy regarding live animals on our premises. Only trained service animals are allowed within our building.