Controlled Experimentation to Guide Product Innovation (Netflix, Groupon, etc.)

Controlled Experimentation (or A/B testing) has evolved into a powerful tool for driving product strategy and innovation. The dramatic growth in online and mobile content, media, and commerce has enabled companies to make principled data-driven decisions. Large numbers of experiments are typically run to validate hypotheses, study causation, and optimize user experience, engagement, and monetization.

The popularity of controlled experimentation has given birth to several companies that specialize in offering tools and services to effectively setup, manage, and monitor large-scale tests. While the simplicity of this concept is disarming, there are several pitfalls that can stymie progress and occasionally lead to poor decisions.

In this panel, we bring together leading experts from the industry to discuss key lessons, challenges, and best practices in online experimentation.

Check out also Rajesh's blog post to frame our discussion: http://www.hivedata.com/controlled-experimentation-to-guide-product-innovation-2/

Meet the experts:

Rajesh Parekh, Senior Director, Data Science (Groupon)

Dr. Rajesh Parekh is Senior Director, Data Science at Groupon where he focuses on applying data mining, machine learning, and optimization algorithms to solving challenging problems in the space of daily deals.

Prior to Groupon, Rajesh was Senior Director of Research at Yahoo! Labs where he led the display advertising targeting sciences focusing on Yahoo!'s flagship Behavioral Targeting algorithms and other online advertising products. At Yahoo! he received the You Rock award for his work on real-time prediction of news-worthy queries, and the Data Wizard award for designing a system to optimize the number of sponsored ads shown on a search results page. Before Yahoo!, Rajesh was at Blue Martini software where he worked in data mining and analytics for retail e-commerce and at Allstate where he tackled key insurance problems such as cross-sell, customer retention, and fraud.

Rajesh earned his Ph.D. in Computer Science from Iowa State University. He has authored over 25 research publications and filed 20 patents. He is actively involved in the data mining community.

Ya Xu, Senior Applied Researcher (Microsoft Bing)

Ya is a Senior Applied Researcher at Microsoft’s Bing where she enjoys applying data mining and statistical techniques to solve challenging practical problems and to guide product development. Much of her work has been focused on improving quality, capacity and efficiency of large-scale online experiments. A sample of her work in these areas can be found in her recent publications in KDD’2012 (http://www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx) and WSDM’2013 (http://www.exp-platform.com/Pages/CUPED.aspx). Prior to Microsoft, Ya earned a PhD in Statistics from Stanford University.

Christian Posse, Program Manager Technology (Google)

Dr. Christian Posse recently joined Google as Program Manager, Technology. Before that he was Principal Product Manager and Principal Data Scientist at LinkedIn where he led the development of recommendation products as well as the next generation online experimentation platform. Prior to LinkedIn, Dr. Posse was a founding member and technology lead of Cisco Systems Network Collaboration Business Unit where he designed the search and advanced social analytics of Pulse, Cisco’s network-based search and collaboration platform for the enterprise. Prior to Cisco, Dr. Posse worked in a wide range of environments, from holding faculty positions in US universities, to leading the R&D at software companies and a US National Laboratory in the social networks, biological networks and behavioral analytics fields. His interests are diverse and include search and recommendation engines, social networks analytics, computational social and behavioral sciences, online experimentation and information fusion. He has written over 40 scientific peer-reviewed publications and holds several patents in those fields. Dr. Posse has a PhD in Statistics from the Swiss Federal Institute of Technology, Switzerland.

Andrew First, CTO (LeanPlum)

Andrew First is CTO at Leanplum, a recent TechStars company focused on optimizing the lifetime value of users in mobile apps. Prior to Leanplum, Andrew worked in video ads optimization at Google and studied Computer Science and Electrical and Computer Engineering at Duke University.

Caitlin Smallwood (Netflix)

Caitlin Smallwood is the Director of Consumer Science and Analysis at Netflix, where she and her team drive experimentation and behavioral analysis to help optimize the Netflix product. With a background in Internet products (Netflix, Intuit, Yahoo!) and analytic consulting (PwC, SRA), Caitlin specializes in analytics, recommendation systems, controlled experiments, product management, business intelligence, and data strategy. Caitlin holds an M.S. in Operations Research from Stanford University and a B.S. in Mathematics from The College of William and Mary.

Agenda:

6.00-7.00pm: registration and networking (pre-party!) with finger food & beverages
7.00-7.40pm: presentations
7.40-8.00pm: Panel Discussion
8.00-8.15pm: Q&A session
8.15-8.45pm: networking session

Directions:

The event will be hosted on the Microsoft campus in Mountain View, located just off Hwy 101 at Shoreline Blvd.

Address: 1065 La Avenida Street, Mountain View, CA 94043. Directions here.

There is construction work on campus so the guest parking is temporarily located at the rear of building 1, between buildings 4 & 5. To access the parking, turn right onto Macon Avenue and follow around past building 5. The temporary lobby entrance is to the left of the guest parking area, at the rear of building 1.