Bangalore Streams In-Person meetup - July 2024

Hello Bengaluru👋
Join us for exciting discussions in the streaming world with opportunities to network with peers and leaders in the industry.

📅 When: July 6 2024 9:30am - 02:00pm
📌Where: ShareChat
🗺Directions: https://maps.app.goo.gl/SywptxCo6Z8hcws6A (ShareChat Mohalla Tech Private Limited , North Tower Smartworks ,Vaishnavi Tech Park, Survey No 16/1 & No 17/2 Ambalipura Village, Varthur Hobli, Bangalore East Taluk – 560103)

🕒 Schedule :
9:30 am - 10:00 am: Welcome & registrations
10:00 am - 10:30 am: Realtime Triggers in Kestra by Shruti Mantri, Software Engineer @ Moveworks
10:35 am - 11:10 am: Moving from Batch to Near-Realtime : Inspired by a true production incident by Shivji Jha, Staff Engineer, Sachidananda Maharana, MTS IV @ Nutanix
11:15 am - 11:50 am: Challenges & Learnings with using Kafka/Redpanda at a huge scale by Anuraj Jain, Software Engineer @ ShareChat
11:50 am - 12:10 pm: Networking break
12:10 pm - 12:45 pm: Unleashing Data Powerhouses - Benthos and Kafka Streams by Shivam Yadav & Shubham Dhal, Software Engineers @ ShareChat
12:45 pm - 1:20 pm: Push Query Layer for Stream Processing Systems (Apache Flink) by Pavan K, CTO @Platformatory & Avinash,Platformatory
1:30 pm - 2:30 pm: Lunch & Networking

🗣Talks:
Realtime Triggers in Kestra
Speaker: Shruti Mantri, Software Engineer at Moveworks
About the talk: 1. Introduction to Kestra
2. Capabilities of Kestra
3. Introduction to Realtime Triggers in Kestra
4. How can realtime triggers be leveraged for event driven architecture

Moving from Batch to Near-Realtime : Inspired by a true production incident
Speaker: Shivji Jha, Staff Engineer @ Nutanix
About the talk: This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don't bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards.

Unleashing Data Powerhouses - Benthos and Kafka Streams
Speaker: Shivam Yadav & Shubham Dhal, Engineers @ ShareChat
About the talk: In this session, we'll unravel the story of ShareChat's transition from Java to Benthos for crafting efficient ETL pipelines. With a single configuration file, Benthos effortlessly connects diverse sources and sinks, transforming the way we handle data. Discover how, armed with "at least once" guarantees, Benthos emerged as the go-to solution for our stateless pipelines.
Learn how we platformized Benthos at ShareChat, deploying over 20 jobs from a single repository. Delve into the specifics of how Benthos with Kafka and Kafka Streams powers our data pipelines, handling tasks such as dumping data to databases, making API calls, and routing events to different message queues. The session will highlight the simplicity and performance benefits we've achieved—processing 2K events/sec to an impressive 45K events/sec—without the need for extensive developer code.

Challenges & Learnings with using Kafka/Redpanda at a huge scale
Speaker: Anuraj Jain, Software Engineer @ ShareChat
About the talk: Kafka is a industry standard data steaming platform and a lot of data intensive companies use it already for their streaming data use-cases, In this talk I am going to cover-up on our (ShareChat's) Challenges & Leaning's from using/operating Kafka/Redpanda at a heavy scale (GBs per sec) and overcoming problems with systems, causes for issues etc. It will be a really good leanings/best-practices session for the devs to avoid mistakes in production at a scale. ShareChat already operates big Redpanda clusters in production and at a heavy scale. I have done a lot of firefighting around various issues and onboarding/migrating of our services/jobs to Kafka protocol recently.

Push Query Layer for Stream Processing Systems
Speaker: Pavan K & Avinash U, Platformatory
About the talk: Stream processing systems mainly work on streaming data and can't hold unlimited state like databases. Some systems allow querying the state store, but this is tricky since state stores are local and not designed for easy access. The talk will explore making state stores queryable in Apache Flink, adding a query layer to integrate Flink into operational workflows.