Bangalore Apache Kafka®, Data and Streams Group

Hello everyone! Join us for an IN PERSON Apache Kafka® meetup on Mar 28 from 11:00AM, hosted by iValue in Bangalore!

📍 Venue:
903/1, iValue Group, 1, 19th Main Rd, opp. Blue Tokai and PurnaBramha Restaurant, Sector 4, HSR Layout, Bengaluru, Karnataka 560102

***
Agenda:

• 11:00 - 11:10: Welcome
• 11:10 - 11:50: Jeevitha Gajendran, Staff Software Engineer, Juniper Networks
• 11:50 - 12:30: Shuva Kar, Senior Principal Engineer, Palo Alto Networks
• 12:30 - 12:40: Break
• 12:40 - 13:20: Animesh Kumar, Senior Software Engineer II, Confluent
• 13:20 - 14:20: Lunch

***
💡 Speaker:
Jeevitha Gajendran, Staff Software Engineer, Juniper Networks

Talk:
Building Real-Time Log Intelligence Pipelines with Kafka & OpenSearch

Abstract:
Modern systems generate massive volumes of logs, but turning them into actionable insights is challenging.

This session explores building a real-time log intelligence pipeline using Kafka and OpenSearch.
Learn how Kafka enables scalable log streaming and OpenSearch powers fast search and analytics.
We’ll walk through an end-to-end architecture and real-world use cases.
By the end, you’ll have a practical blueprint for real-time observability in distributed systems.
-----
💡 Speaker:
Shuva Kar, Senior Principal Engineer, Palo Alto Networks

Talk:
The Agentic Shift in Kafka Stream Processing

Abstract:
Most generative AI applications today rely on a slow, synchronous request-and-response loop, but real-world data doesn't wait to be asked. In this session, we will explore the "Passive Pipeline" architecture—a design pattern that flips traditional LLM integration on its head by using Apache Kafka as the continuous event backbone. We will discuss how to transition from reactive, prompt-based AI to event-driven intelligence, allowing your AI agents to monitor, reason, and react to high-volume data streams in real time without being overwhelmed.
To bring this to life, we will dive into a live, code-heavy demo of a real-time SecOps agent. You will see firsthand how to capture infrastructure events like suspicious IP activity, route them through Kafka, manage localized state, and intelligently feed that stream into Google’s Gemini model for instant threat analysis. Expect practical takeaways on wiring these systems together in Python, overcoming real-world deployment hurdles, and keeping your generative models seamlessly in the loop of a constant data stream.
-----
💡 Speaker:
Animesh Kumar, Senior Software Engineer II, Confluent

Talk:
When Restarts Kill: Solving the Debezium Schema History "Zombie" Problem

Abstract:
What happens when your "self-healing" infrastructure actually makes a production outage worse? In this session, we dissect a high-impact reliability issue where the Kafka Connect operator’s attempt to auto-restart "failed" tasks led to massive thread leaks, CPU spikes, and sometimes even SNAT port exhaustion.

The culprit was a hidden "pathological interaction" within the Debezium MySQL connector. By performing heavy-weight schema history recovery inside the `SourceTask.start()` method, the connector remained in a "failed" state too long. This triggered the operator to restart the task every 60 seconds, creating a "thundering herd" of concurrent recovery threads—or "zombie tasks"—that never terminated.
In this talk, we’ll walk through the journey of identifying this bottleneck and the multi-wave architectural fix (inspired by DBZ-8562). We will cover:

• Decoupling Health from Recovery: How we moved schema history recovery out of the `start()` method to allow tasks to transition to `RUNNING` instantly, preventing the fatal restart loop.
• Silencing the DDL Noise: Strategies for filtering non-essential DDLs—such as Functions, Triggers, and Views—to keep the schema history topic lean and recovery times fast.
• The "Zombie" Thread Phenomenon: Why graceful shutdowns fail during long recoveries and how to protect your cluster from resource exhaustion.

Whether you're a platform engineer or a data practitioner, this session provides a "tale from the trenches" on how subtle interactions between connector internals and orchestration logic define system reliability at scale.