Scaling Kafka in Kubernetes with and without Datadog Kafka-Kit

We will have 3 talks for this meetup! Make sure to bring an ID as one will be required to get into the building.

Scaling your Kafka cluster can be a difficult task. Many companies try to tackle this problem in different ways.

At Capital One, Kafka is used as message broker for multiple real-time streaming systems and has proved very capable. Andreas Markmann will discuss how many data loads scale up slowly over the course of weeks, so that scaling can be addressed via blue/green deployment. Refreshing cloud instances in a running cluster, however, can require rebalancing data over the network, resulting in considerable network load, which can be avoided by reattaching cloud storage. To move toward more seamless scaling of always-on clusters, a flexible solution like Kafka-Kit is desirable.

Datadog has been trying to solve this problem with the introduction of Kafka-Kit. Balthazar Rouberol will give a talk about how this new open source tool works and how it can improve the scalability of your teams kafka clusters in Kubernetes. If you would like to learn more about Kafka-Kit, you can read more about it here: https://www.datadoghq.com/blog/engineering/introducing-kafka-kit-tools-for-scaling-kafka/

Gwen Shapira - from Confluent - will give the third talk:
Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. Kafka Streams is not only scalable, but fully elastic allowing for dynamic scale-in and scale-out as the library handles state migration transparently in the background. By running Kafka Streams applications on Kubernetes, you will be able to use Kubernetes powerful control plane to standardize and simplify the application management—from deployment to dynamic scaling.
In this technical deep dive, we’ll explains the internals of dynamic scaling and state migration in Kafka Streams. We’ll then show, with a live demo, how a Kafka Streams application can run in a Docker container on Kubernetes and the dynamic scaling of an application running in Kubernetes.