align-toparrow-leftarrow-rightbackbellblockcalendarcamerachatcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-crosscrosseditfacebookglobegoogleimagesinstagramlocation-pinmagnifying-glassmailmoremuplabelShape 3 + Rectangle 1outlookpersonplusImported LayersImported LayersImported Layersshieldstartwitteryahoo

Controlled Experimentation to Guide Product Innovation (Netflix, Groupon, etc.)

Controlled Experimentation (or A/B testing) has evolved into a powerful tool for driving product strategy and innovation. The dramatic growth in online and mobile content, media, and commerce has enabled companies to make principled data-driven decisions. Large numbers of experiments are typically run to validate hypotheses, study causation, and optimize user experience, engagement, and monetization.

The popularity of controlled experimentation has given birth to several companies that specialize in offering tools and services to effectively setup, manage, and monitor large-scale tests. While the simplicity of this concept is disarming, there are several pitfalls that can stymie progress and occasionally lead to poor decisions.

In this panel, we bring together leading experts from the industry to discuss key lessons, challenges, and best practices in online experimentation.

Check out also Rajesh's blog post to frame our discussion:

Meet the experts:

Rajesh Parekh, Senior Director, Data Science (Groupon)

Dr. Rajesh Parekh is Senior Director, Data Science at Groupon where he focuses on applying data mining, machine learning, and optimization algorithms to solving challenging problems in the space of daily deals.

Prior to Groupon, Rajesh was Senior Director of Research at Yahoo! Labs where he led the display advertising targeting sciences focusing on Yahoo!'s flagship Behavioral Targeting algorithms and other online advertising products. At Yahoo! he received the You Rock award for his work on real-time prediction of news-worthy queries, and the Data Wizard award for designing a system to optimize the number of sponsored ads shown on a search results page.  Before Yahoo!, Rajesh was at Blue Martini software where he worked in data mining and analytics for retail e-commerce and at Allstate where he tackled key insurance problems such as cross-sell, customer retention, and fraud.

Rajesh earned his Ph.D. in Computer Science from Iowa State University. He has authored over 25 research publications and filed 20 patents. He is actively involved in the data mining community.

Ya Xu, Senior Applied Researcher (Microsoft Bing)

Ya is a Senior Applied Researcher at Microsoft’s Bing where she enjoys applying data mining and statistical techniques to solve challenging practical problems and to guide product development. Much of her work has been focused on improving quality, capacity and efficiency of large-scale online experiments. A sample of her work in these areas can be found in her recent publications in KDD’2012 and WSDM’2013. Prior to Microsoft, Ya earned a PhD in Statistics from Stanford University.

Christian Posse, Program Manager Technology (Google)

Dr. Christian Posse recently joined Google as Program Manager, Technology. Before that he was Principal Product Manager and Principal Data Scientist at LinkedIn where he led the development of recommendation products as well as the next generation online experimentation platform. Prior to LinkedIn, Dr. Posse was a founding member and technology lead of Cisco Systems Network Collaboration Business Unit where he designed the search and advanced social analytics of Pulse, Cisco’s network-based search and collaboration platform for the enterprise. Prior to Cisco, Dr. Posse worked in a wide range of environments, from holding faculty positions in US universities, to leading the R&D at software companies and a US National Laboratory in the social networks, biological networks and behavioral analytics fields. His interests are diverse and include search and recommendation engines, social networks analytics, computational social and behavioral sciences, online experimentation and information fusion. He has written over 40 scientific peer-reviewed publications and holds several patents in those fields. Dr. Posse has a PhD in Statistics from the Swiss Federal Institute of Technology, Switzerland.

Andrew First, CTO (LeanPlum)

Andrew First is CTO at Leanplum, a recent TechStars company focused on optimizing the lifetime value of users in mobile apps. Prior to Leanplum, Andrew worked in video ads optimization at Google and studied Computer Science and Electrical and Computer Engineering at Duke University.

Caitlin Smallwood (Netflix)

Caitlin Smallwood is the Director of Consumer Science and Analysis at Netflix, where she and her team drive experimentation and behavioral analysis to help optimize the Netflix product.  With a background in Internet products (Netflix, Intuit, Yahoo!) and analytic consulting (PwC, SRA), Caitlin specializes in analytics, recommendation systems, controlled experiments, product management, business intelligence, and data strategy.  Caitlin holds an M.S. in Operations Research from Stanford University and a B.S. in Mathematics from The College of William and Mary.


[masked]pm: registration and networking (pre-party!) with finger food & beverages
[masked]pm: presentations
[masked]pm: Panel Discussion
[masked]pm: Q&A session
[masked]pm: networking session


The event will be hosted on the Microsoft campus in Mountain View, located just off Hwy 101 at Shoreline Blvd.

Address: 1065 La Avenida Street, Mountain View, CA 94043. Directions here.

There is construction work on campus so the guest parking is temporarily located at the rear of building 1, between buildings 4 & 5. To access the parking, turn right onto Macon Avenue and follow around past building 5. The temporary lobby entrance is to the left of the guest parking area, at the rear of building 1.

Join or login to comment.

  • A former member
    A former member

    Did Rajesh use slides?

    March 21, 2013

  • Dolly W.

    Speakers were topnotch. Event was well organized.

    1 · March 28, 2013

  • Julia N.

    the audio recording for the talk by Rajesh is available here

    March 26, 2013

  • Alice C.

    Too many speakers. Too little time. Maybe it is better to have just one or two speakers and have more detailed presentation on their analysis.

    1 · March 21, 2013

    • Pashu P.

      thanks for your feedback, taking into account your comment.

      1 · March 21, 2013

  • Pashu P.

    Check out KC Leung's (our favorite photographer) pictures and video who did an awesome job as always!

    March 21, 2013

  • Ellen F.

    Excellent meeting on A/B testing, a topic of particular interest. Good and entertaining speakers with helpful use cases and good questions from audience. Great all around. Thanks!

    1 · March 21, 2013

  • Arun K.

    Hi, I had a great time at the meetup last night. I'd like to thank the organizers (Pashu, Ravi) for putting on a great event I learned quite a bit about A/B testing and made a board with the slides from the event. Our site has over 2.5 million developer resources from over 10K sources. If you'd like to check out the board from last night here you go.

    1 · March 21, 2013

  • Sanjeev S.

    Nice talk!
    Perhaps some of the attendees from the event can mention the commercial tools that they have used for this purpose. Any comments on how well tools like, say, Autonomy Optimost or the cloud-based Optimizely can support the range of controls/dashboards discussed in the talk last night.

    2 · March 21, 2013

  • Inna G.

    This presentation covered very important area of web analytics.
    In general, this meetup has the highest standard of the selected topics, presenters, events organization.

    2 · March 21, 2013

  • Pashu P.

    Slides are now available on Slideshare - Would love to hear your feedback on the event! Thanks

    4 · March 20, 2013

    • Shawn

      Is the Groupon slides available as well?

      March 21, 2013

  • Melisa

    This was a fantastic Meetup! Informative, lively, with just enough content & discussion to make you want more. I do agree I wish there were more time for Q&A but I thought the number of presenters, their caliber and personal styles and their content was well worth the time.

    2 · March 21, 2013

  • KC L.

    Fantastic speakers, I learn so much at every single of the Hive Meetup ! This has got to be the best Meetup Group in Silicon Valley, tasty appetizers and fine wine ! Its always a joy to attend !

    2 · March 20, 2013

  • Sam K.

    Great turnout for networking and enjoyed the informative and lively panel discussion on controlled experimentation. I learned a lot about a subject that I knew very little about .....

    3 · March 20, 2013

  • A former member
    A former member

    Unfortunate time conflict

    March 20, 2013

  • Krishna

    Apologize for the late notice.

    March 20, 2013

  • Leslie P.

    Interested in connecting with folks interested in rapid prototyping for testing and lean development.

    March 20, 2013

  • Suyash J.

    Unfortunately I won't be able to make this time, can you please repeat this talk again perhaps in City ?

    March 20, 2013

  • Parag P.


    March 20, 2013

  • T. M. R.

    Rajesh Parekh of Groupon frames the discussion in this new blog post:

    2 · March 19, 2013

    • Cheong A.

      Thanks for the informative article! After reading this, I was wondering how to set up A/B Tests on my server (Linux plus mostly open-sourced software) without developing new tools, which I can do as a web developer+data analytics person. Googling around revealed a couple of interesting open-source/free tools I can understand:­

      I am not related to any of the developers above. Just want to share what I found.

      1 · March 20, 2013

  • Amory S.

    Can anyone offer a lift to or from SF? Would love to go, but it's a long trek by transit...

    1 · March 14, 2013

    • A former member
      A former member


      March 18, 2013

    • Yan

      I could pick up three people. You can reach me at @ outlook. Will be leaving from Mission area.

      March 19, 2013

  • Ronald Z.

    occupied with something

    March 19, 2013

  • Ellen F.

    Interesting topic and strong combination of speakers - should be very rewarding evening.

    3 · March 19, 2013

    • Pashu P.

      Thank you Ellen! We very much look forward to seeing you there. Sorry you missed our 'Harlem Shake' last Wednesday. Hope you had a great event.

      March 19, 2013

  • Prasanth M.

    Sounds interesting

    1 · March 19, 2013

  • Filomena U

    Looking forward to some great learning!

    1 · March 19, 2013

  • Sam K.

    Looking forward to this meetup !!!

    1 · March 19, 2013

  • Sridhar R.

    Sounds very interesting!

    1 · March 19, 2013

  • Melisa

    I'm driving in from the Monterey area, is anyone carpooling from this far South?

    March 18, 2013

  • A former member
    A former member

    Startup kids ready to learn

    1 · March 18, 2013

  • A former member
    A former member

    Apologies - out of country

    1 · March 11, 2013

Our Sponsors

  • Visa

    Global payments technology company.

  • Pentaho

    Reporting, analysis, dashboard, data mining and workflow capabilities.

  • Aerospike

    Flash-optimized in-memory open source NoSQL database.

  • Teradata

    Teradata is a provider of enterprise analytic technologies and services.

  • Cloudera

    Enterprise Platform for Big Data. Leading Solution for Apache Hadoop.

  • Foghorn

    Multi-tier deployment platform for data-driven IoT applications.

  • Perkins Coie

    International firm specializing in business law & litigation.

  • O'Reilly Strata

    Big data technology and strategy Conference.

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy