Stream Processing with Apache Kafka & Apache Samza

Welcome to the upcoming Stream Processing (Virtual) Meetup hosted by LinkedIn! This meetup focuses on Apache Kafka, Apache Samza, and related streaming technologies.

Location: https://linkedin.zoom.us/j/94665364573

Agenda:

6:00 - 6:05 PM: Welcome & Introductions

6:05 - 6:45 PM: Data Integration Platform using Brooklin
Santosh Domalapalli, Wayfair

Wayfair’s rapid growth resulted in separate and specialized platforms and solutions to move data among heterogeneous data systems. These systems became hard to manage and scale, and their workflows perpetuated data governance anti-patterns.

In this presentation, we will discuss how Wayfair is rationalizing and consolidating the data movement solutions under one Data Integration Platform, powered by Brooklin. We will discuss how Wayfair extended Brooklin to address the challenges of streaming Change Data Capture and Domain Events from production databases and applications into Kafka, Google Cloud Storage and Google BigQuery to support various real-time applications and analytical use cases.

6:45 - 7:25PM: Exploring Stream-to-Batch Unification at LinkedIn
Xinyu Liu & Yuhong Cheng, LinkedIn

Running a single program for both batch and stream processing has been the emerging requirement for many use cases at LinkedIn. These requirements pose a major challenge to our existing data infra. In streaming, Apache Samza has been powering thousands of applications to process 2 trillion messages daily with large states and fault tolerance. In batch, we use Apache Spark to solve sophisticated batch scenarios and process PBs of data with our industry-leading external shuffling service and schema metadata store. To enable unified data processing, we converge these two powerhouses by leveraging Apache Beam API. In this talk, we will go through the challenges of running the unified pipelines, the progress we made, and the lessons learned.

Want to talk at a future meetup?
Please contact us via the “Contact” button in meetup.com.

The Streams team is hiring!
SWE: https://bit.ly/3wVDDjF
Senior SWE: https://bit.ly/2U4tEtS
Staff SWE: https://bit.ly/2U21z6o