Name: Spark Streaming and GraphX at Netflix
Start: 2015-05-19T18:30:00-07:00
End: 2015-05-19T21:30:00-07:00
Location: Netflix

This meetup will be at Netflix and feature talks by Netflix engineers about their use of Spark Streaming and GraphX, as well as a Q&A session with the Netflix folks plus the lead engineer of Spark Streaming. Find detailed abstracts for both talks below.

Agenda for the evening:
6:30 Mingling
7:00 Two Netflix talks
8:15 Q&A (Prasanna, Bharat, Yves , Ehtsham, and TD)
8:30 Mingling
9:00 End

The talks will be live streamed and the video will be published on the Apache Spark channel (https://www.youtube.com/playlist?list=PL-x35fyliRwiP3YteXbnhk0QGOtYLBT3a) on YouTube.

See this Netflix tech blog post (http://techblog.netflix.com/2015/03/can-spark-streaming-survive-chaos-monkey.html) for a primer.

Talk #1: Spark Streaming ResiliencyBy Prasanna Padmanabhan and Bharat Venkat, Senior Software Engineers

Netflix is a data-driven organization that places emphasis on data quality, availability and agility to capture and process that data. Some of our recommendation algorithms are computed as events happen in real time. Such streaming applications are long running tasks that need to be resilient. This is especially true in a cloud deployment due to the ephemeral nature of resources. In this talk, we will cover the What, the Why and the How of our resiliency exercise with Spark Streaming in an AWS cloud deployment. A Netflix ChaosMonkey based approach, which randomly terminated instances or processes, was employed to simulate failures. We hope that this exercise will help build confidence in the resiliency on Spark Streaming for similar contexts.

Talk #2: Spark and GraphX in the Netflix Recommender SystemBy Yves Raimond and Ehtsham Elahi, Senior Research Engineers

We at Netflix strive to deliver maximum enjoyment and entertainment to our millions of members across the world. We do so by having great content and by constantly innovating on our product. A key strategy to optimize both is to follow a data-driven method. Data allows us to find optimal approaches to applications such as content buying or our renowned personalization algorithms. But, in order to learn from this data, we need to be smart about the algorithms we use, how we apply them, and how we can scale them to our volume of data (over 60 million members and 10 billion hours streamed last quarter). In this talk we describe how Spark and GraphX can be leveraged to address some of our scale challenges. In particular, we share insights and lessons learned on how to run large probabilistic clustering and graph diffusion algorithms on top of GraphX, making it possible to apply them at Netflix scale.

Andy Konwinski

Faisal Zakaria Siddiqi

Bay Area Spark Meetup

Technology

Apache Spark

Big Data

Machine Learning

Hadoop

Data Analytics

Open Source

Scala

Cloud Computing

Functional Programming

MapReduce

Hive

Artificial Intelligence

Spark Streaming and GraphX at Netflix

Netflix

Share

Bay Area Spark Meetup

Spark Streaming and GraphX at Netflix

Bay Area Spark Meetup

Details

Members are also interested in