Apache Kafka® with Dropbox + Ending endless rebalances @ Confluent's new MV HQ

Join us for an Apache Kafka® meetup on December 5th at 5:30pm, hosted at Confluent's brand new office and event space in Mountain View! All details below

ATTEND

1. RSVP below
2. Fill in this short form: http://bit.ly/32A4xh7
3. Prior to the event, you will receive an email asking you to register for the event and sign an NDA, if you do that we'll have a badge ready for you when you arrive!

----

5:30pm: Networking, Pizza and drinks!
6:00pm: Guozhang Wang and Sophie Blee-Goldman, Confluent
6:40pm: Peng Kang and Richi Gupta, Dropbox
7:20-8pm: Additional Q&A and Networking

---

First Talk: The Silver Bullet for Endless Rebalances

Abstract:
Anyone who has used Kafka consumer groups or operated a Kafka Streams application is likely familiar with the rebalancing protocol, which is used to (re)distribute partitions among the consumers of a group whenever there is a change in membership or in the topics subscribed to. The current protocol takes the safest possible approach of pausing all work and revoking ownership of all partitions so that a new assignment can be made. This “stop-the-world” approach can be frustrating especially when the mapping of partitions to the consumer that owns them barely changes. In KIP-429 we introduce incremental cooperative rebalancing for the consumer client, a new rebalancing protocol that allows consumers to retain ownership and continue fetching for their owned partitions while a rebalance is in progress. This proposal trades extra rebalances for the ability to revoke only those partitions which are to be migrated to another consumer for overall workload balance.

Bios:
Sophie Blee-Goldman joined the Streams team at Confluent just under a year ago. She has been continuing to work on rebalancing improvements in the Kafka consumer and the Streams layer.

Guozhang Wang is a PMC member of Apache Kafka, and also a tech lead at Confluent leading the Streams team. He receives his PhD from Cornell University database group where he worked on scaling iterative data-driven applications. Prior to Confluent, Guozhang was a senior software engineer at LinkedIn, developing and maintaining its backbone streaming infrastructure on Apache Kafka and Apache Samza.

--

Second Talk: Kafka as a service at Dropbox

Abstract:
As a scalable and reliable data streaming solution with a rich ecosystem, Kafka is widely adopted in Dropbox infrastructure in various scenarios. It is part of Dropbox’s analytics data pipeline, stream processing platform and more mission critical systems. Jetstream is the team that provides Kafka as a service in Dropbox infrastructure. We manage the clusters, develop tooling, and enforce policies, so that our users can enjoy a highly available and reliable service. In this talk, we will share our experiences and learnings running Kafka clusters, pipelines that enable high durability (direct writes to kafka) and availability (goscribe), the policies we enforce for high reliability, the tooling we have for maintenance and stress testing, and finally an overview of Dropbox’s next generation queueing service built on top Kafka.

Bios:
Peng Kang is a software engineer from the Jetstream team at Dropbox. Before joining Jetstream, Peng was a main contributor to Cape, which is Dropbox’s in-house ultra-low latency stream processing platform. Peng has extensive experience in architecting and developing large scale distributed infrastructure systems.

Richi Gupta is Engineering Manager of the Jetstream team at Dropbox. Prior to joining Dropbox, Richi spent several years at Intel in various positions culminating as software engineering manager. She led a team of 20+ engineers across 3 countries. Her team developed advanced routing algorithms based on computational geometry to automate layout generation for Intel’s chip design teams.

Slack: https://launchpass.com/confluentcommunity

NOTE: We are unable to cater for any attendees under the age of 21.