Skip to content

Details

Welcome to the upcoming Stream Processing (Virtual) Meetup hosted by LinkedIn! This meetup focuses on Apache Kafka, Apache Samza, and related streaming technologies.

Location: Bluejeans Primetime
https://primetime.bluejeans.com/a2m/live-event/kwzghqyr

Agenda:

6:00 - 6:05 PM: Welcome & Introductions

6:05 - 6:45 PM: Beam Samza Runner performance benchmarking and improvements
Yixing Zhang & Bingfeng Xia, LinkedIn

Since Beam Samza runner was developed in 2018 at LinkedIn, it has been widely adopted at LinkedIn for complex stream processing use cases such as fraud detection, metrics reporting, and feature generation for machine learning. We now have 100+ Samza Beam jobs running in production, and as adoption continues to quickly rise, we strived to understand: a) How Samza runner performs compared to other Beam runners? b) Where is the performance bottleneck and how to improve it? In this presentation, we will talk about how we utilized Beam Nexmark test suite to benchmark the performance of Beam runners. We will also share with you how we used a CPU async profiler to discover performance bottlenecks for Samza runner, and how we successfully improved the average Nexmark throughput of Samza runner by more than 10 times! With higher throughput, we can scale Samza in a more efficient way and hence reduce the cost.

6:45 - 7:30PM: Faster and Safer Cluster Maintenance with Cruise Control's Concurrency Adjuster
Efe Gencer, LinkedIn

Cruise Control has enabled the automated management of large-scale Kafka clusters. Typical management operations involve detection and remediation of (1) failed or slow brokers/disks, (2) load imbalance, (3) SLO violations, and (4) misconfigured topic properties. Yet existing remediation processes either take a long time to complete or exhibit user-visible side effects unless there is human supervision to oversee the process with constant evaluation and feedback. This talk introduces a new automated component, called Concurrency Adjuster, which enables the execution of remediation actions as fast and safely as possible. The key insight that allows Concurrency Adjuster to ensure a reliable speedup is a feedback loop that uses the relevant broker metrics to adaptively adjust the execution pace.

Want to talk at a future meetup?
Please contact us via the “Contact” button in meetup.com.

You may also like