Running big Apache Kafka® upgrades at Yelp + Multi-tenant Performance Protection

Join us for an Apache Kafka® meetup on February 20th at 5:30pm, hosted at Confluent's brand new office and event space in Mountain View! All details below

ATTEND

1. RSVP below
2. Fill in this short form: https://docs.google.com/forms/d/e/1FAIpQLSfS2LCA1T6rbnE9QXVzcEHoVm-Kvo6Z6sAJZkSL_VTw84vSCA/viewform
3. Prior to the event, you will receive an email asking you to register for the event and sign an NDA, if you do that we'll have a badge ready for you when you arrive!

You can still attend if you do not fill the form above, though it may take slightly longer to check in.

----

5:30pm: Networking, Pizza and drinks!
6:00pm: Anna Povzner, Confluent
6:40pm: Manpreet Singh, Yelp
7:20-8pm: Additional Q&A and Networking

---

First Talk: Anna Povzner

Protecting Tenant Performance in Multi-tenant Kafka

Abstract:
Deploying Kafka to support multiple teams or even an entire company has many benefits. It reduces operational costs, simplifies onboarding of new applications as your adoption grows, and consolidates all your data in one place. However, this makes applications sharing the cluster vulnerable to any one or few of them taking all cluster resources. The combined cluster load also becomes less predictable, increasing the risk of overloading the cluster and data unavailability.

In this talk, we will describe how to use quota framework in Apache Kafka to ensure that a misconfigured client or unexpected increase in client load does not monopolize broker resources. You will get a deeper understanding of bandwidth and request quotas, how they get enforced, and gain intuition for setting the limits for your use-cases.

Bio:
Anna Povzner is a software engineer on Cloud Native Kafka team at Confluent, and a contributor to Apache Kafka. Her main area of expertise is in resource management for performance SLAs and multi-tenancy in storage and distributed data systems. She received her Ph.D. from U.C. Santa Cruz, and was a researcher at IBM Almaden. Prior to Confluent, she was one of the early engineers in a storage startup where she helped build a scale-out content-addressable storage system from scratch.

--

Second Talk: Manpreet Singh

Running Large Scale Kafka Upgrades at Yelp

Abstract: Over the years at Yelp, we have relied on Kafka to build many complex applications and stream processing data-pipelines that solve a multitude of use cases, including powering our product experimentation workflow, search indexing, asynchronous task processing and more. This session will focus on the challenges we encountered and how we evolved our infrastructure tooling and upgrade strategy to overcome them. I will be talking about: — How we rolled out new features such as kafka offset storage, message timestamp, reassignment auto-throttling, etc. — Core technical issues discovered during upgrades such as failure of log cleaners due to large offsets while upgrading. — The in-house test-suite that we built in order to: validate new kafka versions against our existing tooling and client-libraries, exercise the upgrade and rollback process and benchmark performance. — The automation we built for safe and fast rolling upgrades and broker configuration deployment.

Bio:
Manpreet Singh is a Software Engineer at Yelp who has led multiple infrastructure projects. He and his teammates are responsible for building and maintaining Yelp’s streaming and stream processing infrastructure using Kafka, handling billions of messages and terabytes of data per day. In his time at Yelp, he has designed, built and automated tooling for Kafka cluster rebalancing and has spearheaded the efforts to upgrade Kafka clusters and isolate revenue critical traffic to dedicated clusters. Manpreet is currently working on rearchitecting the deployment of Kafka Clusters to better cater to Yelp’s ever-growing scale.

Slack: https://launchpass.com/confluentcommunity

Want to talk at a future meetup?
Please contact community@confluent.io