Streaming in the Real World: Reliability Lessons from Kafka and WarpStream

Join us for an Apache Kafka® meetup on November 19th from 6:00pm hosted by Salesforce!

📍 Venue:
Salesforce
333 Seymour St #700, Vancouver, BC V6B 5A7
Social Lounge

IMPORTANT: Please arrive on time so that we can check you in, many thanks!

🗓 Agenda:

• 6:00pm - 6:30pm: Doors Open, Networking, Pizza and Drinks
• 6:30pm - 7:00pm: Dan O'Reilly, Senior Software Engineer, Salesforce
• 7:00pm - 7:30pm: Ryan Belgrave, Staff Software Engineer I, WarpStream
• 7:30pm - 8:30pm: More networking, Q&A

💡 Speaker One:
Dan O'Reilly, Senior Software Engineer, Salesforce

Talk:
Stability Through Static Membership

Abstract:
Join me for a deep dive into how we keep our large-scale Kafka and Kafka Connect pipelines stable at Salesforce — even during frequent Kafka releases and node patching.

In this talk, we’ll cover:

• Our Kafka Architecture: A high-level look at our Kafka and Kafka Connect setup and how it supports our growing pub/sub use cases.
• Scaling Pains: Real-world challenges we’ve faced with availability, consumer group reliability, and task-level rebalancing.
• Exploring Solutions: Key KIPs and other approaches we evaluated to improve stability.
• Static Membership in Action: How we implemented Static Membership for task-level consumer groups in Kafka Connect.
• Lessons Learned & Improvements: Practical insights from our deployment, including tweaks and optimizations that made a real difference.

Whether you’re running large Kafka clusters or just curious about improving consumer group stability, this talk shares practical takeaways you can apply right away.

💡 Speaker Two:
Ryan Belgrave, Staff Software Engineer I, WarpStream

Talk:
From Ticker Tape to Trendlines: A Stream Processing Journey into Market Dynamics

Abstract:
Building a platform to analyze real-time market data can be a complex undertaking. This session details an end-to-end project for ingesting, processing, and visualizing data from an active and entirely digital marketplace. We'll focus on moving beyond simple metrics to uncover deeper economic trends and behaviors.

Here’s what I’ll be covering:

• Tapping the Data Firehose: I’ll explain how to use public APIs—the source of which might surprise you—to ingest a stream of real-time trading events, as well as how to effectively backfill years of historical data to get a complete market picture.
• Building a Pipeline with Bento and WarpStream: I’ll demonstrate how to use Bento to seamlessly capture this trading data and publish it into WarpStream, a diskless, Kafka-compatible streaming platform designed for the cloud.
• From Streams to Lakehouse with Tableflow and Iceberg: I'll showcase how WarpStream's Tableflow feature automatically materializes our streaming data directly into an Apache Iceberg data lakehouse.
• Visualizing the Market with Grafana: The processed data is then brought to life in Grafana. I will show how to build various dashboards to track price histories, perform complex aggregations, and create custom "market indexes" to gauge the overall health of the economy.
• The Benefits of a Modern Streaming Lakehouse: I’ll explain the technical reasons for choosing this specific stack and how its storage-based architecture helped build a powerful analytics system while minimizing complexity and cost.

If you're interested in stream processing, data engineering, analytics, or are just curious to see what it takes to analyze a complex and fascinating market, this session is not one to miss!

Bio:
Ryan Belgrave’s journey in technology began in 2005 with his first Ubuntu CD and a struggle to exit Vim, igniting a passion for learning technologies like Openstack, AWS EC2, Mesos, Docker, and Kubernetes. In 2016, he joined Target to build their Cloud Application platform. Later, at Optum, he developed a Cluster as a Service for Cassandra, Kafka, and Elasticsearch, managing thousands of clusters with over 12PB of data. In late 2024, he joined Confluent to expand their WarpStream zero-disk Kafka product. He also maintains a homelab to continue learning outside of work.

***
DISCLAIMER
NOTE: We are unable to cater for any attendees under the age of 18.
If you would like to host or speak at a meetup, please email community@confluent.io