Offset Management, Netflix Data Pipeline, Best Practices


Details
Come join us for food, drinks and lots of discussion on Apache Kafka!
Live streaming at http://www.ustream.tv/linkedin-events
Chat on Freenode IRC in #apache-kafka for remote Q&A: https://webchat.freenode.net/?channels=#apache-kafka
Agenda:
• Networking - 6:00PM
• Opening - 6:30 PM - Clark Haskins (LinkedIn)
• Offset management - 6:35 PM - Joel Koshy (LinkedIn)
• The Netflix Data Pipeline - 7:05PM - Allen Wang & Steven Wu (Netflix)
• Best Practices - 7:50PM - Jay Kreps (Confluent)
• Q&A - 8:20PM - ???
Offset Management
Kafka 0.8.2 introduces a new offset management feature which enables consumers to store their offsets in a special compacted Kafka topic. In this talk, we will discuss the motivation behind this approach, how it works, how to use it and how to monitor its usage in production.
The Netflix Data Pipeline
We are going to give a high level overview of Netflix data pipeline, as well as why and how we migrate to use Kafka. We will discuss in detail how we deploy, operate and scale Kafka in AWS cloud and the lessons learned.
Best Practices for a Kafka-based Stream Data Platform
This talk will cover some of the best practices we have seen from Kafka users in how to build out streaming data collection and processing in their organizations. This will include dealing with data formats and schemas and special tips for working with different types of data and integrating with different data systems.

Sponsors
Offset Management, Netflix Data Pipeline, Best Practices