Skip to content

Offset Management, Netflix Data Pipeline, Best Practices

Photo of Jon Bringhurst
Hosted By
Jon B. and Clark Elliott Haskins I.
Offset Management, Netflix Data Pipeline, Best Practices

Details

Come join us for food, drinks and lots of discussion on Apache Kafka!

Live streaming at http://www.ustream.tv/linkedin-events

Chat on Freenode IRC in #apache-kafka for remote Q&A: https://webchat.freenode.net/?channels=#apache-kafka

Agenda:

• Networking - 6:00PM

• Opening - 6:30 PM - Clark Haskins (LinkedIn)

• Offset management - 6:35 PM - Joel Koshy (LinkedIn)

• The Netflix Data Pipeline - 7:05PM - Allen Wang & Steven Wu (Netflix)

• Best Practices - 7:50PM - Jay Kreps (Confluent)

• Q&A - 8:20PM - ???

Offset Management

Kafka 0.8.2 introduces a new offset management feature which enables consumers to store their offsets in a special compacted Kafka topic. In this talk, we will discuss the motivation behind this approach, how it works, how to use it and how to monitor its usage in production.

The Netflix Data Pipeline

We are going to give a high level overview of Netflix data pipeline, as well as why and how we migrate to use Kafka. We will discuss in detail how we deploy, operate and scale Kafka in AWS cloud and the lessons learned.

Best Practices for a Kafka-based Stream Data Platform

This talk will cover some of the best practices we have seen from Kafka users in how to build out streaming data collection and processing in their organizations. This will include dealing with data formats and schemas and special tips for working with different types of data and integrating with different data systems.

Photo of Bay Area Apache Kafka® Meetup group
Bay Area Apache Kafka® Meetup
See more events