Name: Streaming in the Real World: Reliability Lessons from Kafka and WarpStream
Start: 2025-11-19T18:00:00-08:00
End: 2025-11-19T21:00:00-08:00
Location: Salesforce Vancouver Office - GO TO THE 7th FLOOR TO SIGN IN

Join us for an Apache Kafka® meetup on November 19th from 6:00pm hosted by Salesforce!

**📍 Venue:**
Salesforce
333 Seymour St #700, Vancouver, BC V6B 5A7
Social Lounge

**IMPORTANT: Please arrive on time** so that we can check you in, many thanks!

**🗓 Agenda:**

* 6:00pm - 6:30pm: Doors Open, Networking, Pizza and Drinks
* 6:30pm - 7:00pm: Dan O'Reilly, Senior Software Engineer, Salesforce
* 7:00pm - 7:30pm: Ryan Belgrave, Staff Software Engineer I, WarpStream
* 7:30pm - 8:30pm: More networking, Q&A

**💡 Speaker One:**
Dan O'Reilly, Senior Software Engineer, Salesforce

**Talk:**
Stability Through Static Membership

**Abstract:**
Join me for a deep dive into how we keep our large-scale Kafka and Kafka Connect pipelines stable at Salesforce — even during frequent Kafka releases and node patching.

In this talk, we’ll cover:

* **Our Kafka Architecture**: A high-level look at our Kafka and Kafka Connect setup and how it supports our growing pub/sub use cases.
* **Scaling Pains**: Real-world challenges we’ve faced with availability, consumer group reliability, and task-level rebalancing.
* **Exploring Solutions**: Key KIPs and other approaches we evaluated to improve stability.
* **Static Membership in Action**: How we implemented Static Membership for task-level consumer groups in Kafka Connect.
* **Lessons Learned & Improvements**: Practical insights from our deployment, including tweaks and optimizations that made a real difference.

Whether you’re running large Kafka clusters or just curious about improving consumer group stability, this talk shares practical takeaways you can apply right away.

**💡 Speaker Two:**
Ryan Belgrave, Staff Software Engineer I, WarpStream

**Talk:**
From Ticker Tape to Trendlines: A Stream Processing Journey into Market Dynamics

**Abstract:**
Building a platform to analyze real-time market data can be a complex undertaking. This session details an end-to-end project for ingesting, processing, and visualizing data from an active and entirely digital marketplace. We'll focus on moving beyond simple metrics to uncover deeper economic trends and behaviors.

**Here’s what I’ll be covering:**

* **Tapping the Data Firehose:** I’ll explain how to use public APIs—the source of which might surprise you—to ingest a stream of real-time trading events, as well as how to effectively backfill years of historical data to get a complete market picture.
* **Building a Pipeline with Bento and WarpStream:** I’ll demonstrate how to use Bento to seamlessly capture this trading data and publish it into WarpStream, a diskless, Kafka-compatible streaming platform designed for the cloud.
* **From Streams to Lakehouse with Tableflow and Iceberg:** I'll showcase how WarpStream's Tableflow feature automatically materializes our streaming data directly into an Apache Iceberg data lakehouse.
* **Visualizing the Market with Grafana:** The processed data is then brought to life in Grafana. I will show how to build various dashboards to track price histories, perform complex aggregations, and create custom "market indexes" to gauge the overall health of the economy.
* **The Benefits of a Modern Streaming Lakehouse:** I’ll explain the technical reasons for choosing this specific stack and how its storage-based architecture helped build a powerful analytics system while minimizing complexity and cost.

If you're interested in stream processing, data engineering, analytics, or are just curious to see what it takes to analyze a complex and fascinating market, this session is not one to miss!

**Bio:**
Ryan Belgrave’s journey in technology began in 2005 with his first Ubuntu CD and a struggle to exit Vim, igniting a passion for learning technologies like Openstack, AWS EC2, Mesos, Docker, and Kubernetes. In 2016, he joined Target to build their Cloud Application platform. Later, at Optum, he developed a Cluster as a Service for Cassandra, Kafka, and Elasticsearch, managing thousands of clusters with over 12PB of data. In late 2024, he joined Confluent to expand their WarpStream zero-disk Kafka product. He also maintains a homelab to continue learning outside of work.

\*\*\*
DISCLAIMER
NOTE: We are unable to cater for any attendees under the age of 18.
If you would like to host or speak at a meetup, please email community@confluent.io

Alice Richardson

Apache Kafka - Vancouver, BC

Confluent

Technology

Open Source

Stream Processing

Apache Kafka

Streaming in the Real World: Reliability Lessons from Kafka and WarpStream

Salesforce Vancouver Office - GO TO THE 7th FLOOR TO SIGN IN

Share

Apache Kafka - Vancouver, BC

Streaming in the Real World: Reliability Lessons from Kafka and WarpStream

Apache Kafka - Vancouver, BC

Details

Members are also interested in