Monitoring is essential to understand the behaviour of a Kafka ecosystem. Although tracing is one of the three pillars of observability, we rarely see it in action with our clients. With the help of traces, we can extract knowledge about the topics a Kafka producer writes to and the topics a Kafka consumer reads from, thereby gaining a deeper understanding of our data pipelines.At this meetup, we will explore the possibilities and limitations of monitoring solutions through case studies, including dead letter queue handling, computation of aggregations with Kafka Streams, message duplication and data loss. As this meetup is a joint venture with Datadog, we will receive support from a monitoring specialist and gain insight from the Datadog monitoring platform.
Agenda:
- introduction to distributed tracing
- show-cases with an open-source monitoring solution
- show-cases with Datadog
- Food and Drinks
This event is partly sponsored by Datadog! Thanks for that!
