4 Meetups and a Conference, or the Road to Current 2024 (session one)

Hello Streamers!

This summer, we’d like to cordially invite everyone to join us in the Bay Area for a series of four meetups leading up to Current.

Please join us for our first meetup in this exciting series on Thursday, May 30th, 5:30pm. Space is limited.

• In June, we will discuss the Kafka ecosystem
• Moving into July, we will have a Flink meetup!
• And lastly, our farewell to the summer series will be a Kafka and Analytics meetup.

📍Venue:
Confluent Rooftop Patio!
899 W Evelyn Ave
Mountain View, CA
5th floor (In the event of inclement weather, we will meet in the Cloud Cafe, 1st floor behind the front desk)

****Please note******: It will be required for all attendees to sign an NDA upon arrival to the meetup. If you would like to sign in advance of the meetup, please complete this form. Very important to note: your email address will only be used to send you an Envoy invite for advance visitor registration to Confluent HQ.

***
🗓 Agenda:

• 5:30pm: Doors Open/food and drinks
• 6:00pm-6:45pm:Uber's Kafka Ecosystem for Enhanced Observability, Andy Han, Uber
• 6:45pm - 7:30pm: Kafka Transactions, Justine Olshan, Sr. Software Engineer II, Confluent
• 7:30pm-8:00pm:Additional Q&A and Networking

***
💡 Speaker for first talk:
Andy Han, Staff Software Engineer, Uber

Title of Talk:
UATU: Navigating the Data Highway - Uber's Kafka Ecosystem for Enhanced Observability

Abstract:
Kafka stands as the backbone of Uber's real-time data architectures, serving as the essential data highway for processing ~12 trillion messages daily. Its critical role in ensuring fast, reliable, and real-time data processing for ride and delivery services, data ingestion, logging, change data capture, and pub/sub makes it indispensable. However, with the increasing scale and complexity of Uber's Kafka ecosystem, achieving smooth operation and user satisfaction has become a formidable challenge. Strategic, reliable, and fine-grained observability is not just necessary but vital to navigating the intricacies of Kafka, providing accurate insights, and addressing issues promptly to uphold the core goals of Uber's business operations.

In this talk, we will present our end-to-end Kafka auditing/monitoring platform, a.k.a Uatu, for tackling the observability problems. On a high level, it provides a unified/single pane of view of the SLA breaches ( key SLAs including data durability, e2e latency, availability, and retention) in this platform, and also fine-grained dissection of the system for issue isolation. The audience can learn from the design philosophy we adopted to systematically monitor a complex system, and how we make tradeoffs between accuracy, timeliness and cost based on factors such as topic criticality to accommodate different user requirements.

Specifically, we will cover the following topics:

• Overview of the Kafka Ecosystem at Uber, and what makes the observability problem challenging?
• Uatu architecture and design deep dive
• Practical use cases (e.g. tracking e2e message loss) that we have, and pain points that we solve for users.
• Operational experiences/lessons, and future roadmap

Bio:
Qiushi (preferred name: Andy) Han is a Staff Software engineer working at Uber. His interests include cloud computing, software architecture, and distributed system design. He currently leads a few projects focusing on improving the scalability, reliability, and observability of the Kafka Ecosystem at Uber.

***
💡 Speaker for second talk:
Justine Olshan, Sr. Software Engineer II, Confluent

Title of Talk:
Kafka Transactions

Abstract:
One crucial part of using transactions in Apache Kafka is reading the result with a read committed consumer! This consumer does the extra work to filter out the messages that are part of an aborted transaction, so they are not returned to the user. In this talk, we will trace the messages through the transaction and explain how the consumer filtering works and why it is implemented this way. We will also touch upon ongoing KIPs that relate to read committed consumers.

Bio:
Justine is a software engineer at Confluent. She has been working on Apache Kafka since 2019. She has worked the on producer, topic IDs, and most recently exactly once semantics. Justine became a committer in 2022 and a PMC member in 2023.

***
DISCLAIMER
BY ATTENDING THIS EVENT IN PERSON, you acknowledge that risk includes possible exposure to and illness from infectious diseases including COVID-19, and accept responsibility for this, if it occurs.
NOTE: We are unable to cater for any attendees under the age of 21.
***