Skip to content

Details

Welcome to the upcoming Stream Processing (Virtual) Meetup hosted by LinkedIn! This meetup focuses on Apache Kafka, Apache Samza, Apache Flink, and related streaming technologies.

Location: https://linkedin.zoom.us/j/91989084874?pwd=OU1BblRFTGR3WmExOXdYZVdGNmlCZz09

Agenda:

6:00 - 6:05 PM: Welcome & Introductions

6:05 - 6:45 PM: AI Flow: A Unified Workflow for Batch and Streaming
Jiangjie (Becket) Qin, Alibaba

Abstract: The concept of workflow is widely recognized in the industry. There are also popular projects such as Azkaban, AirFlow, KubeFlow helping people build and run workflows that consist of various jobs. However, it is really interesting that stream processing jobs have rarely been working well with workflows, mainly because streaming jobs do not run to finish. This talk introduces a project called AI Flow addressing the problem. It helps people define and run workflows containing both batch and stream jobs. The talk will show how AI Flow can help the users obtain a global view and overall control of their applications. Although AI Flow initially grew from the real time machine learning use case in the Apache Flink community, it can be applied to much broader scenarios.

  • Bio: Jiangjie (Becket) Qin is a software engineer leader at Alibaba Group, focusing on Apache Flink. Prior to that, Becket worked at LinkedIn to build streams infrastructures around Apache Kafka. Becket is an ASF Member, PMC member of Apache Flink and Apache Kafka.

6:45 - 7:25 PM: Auto-sizing for Stream Processing Applications at LinkedIn
Ray Matharu Singh and Ziting Shen, LinkedIn

Abstract: Samza provides a stream processing platform-as-a-service (PaaS) offering at LinkedIn, which is used to host thousands of business-critical applications from Notifications, Trust, Abuse, InSearch, Ads, Feed, among others. This requires service owners to manage applications' resource sizing and tuning. Unfortunately, applications have evolved from their conventional model of a directed acyclic graph of operators and incorporate multiple other functionalities, which presents numerous challenges for sizing. We present our work on Samza Autosizing, a controller that dynamically controls applications' resource sizing while accounting for diverse functionalities, input load variations, and remote service dependencies (such as Venice, Espresso, RestLi), to maximize cluster utilization and minimize cost. We discuss the challenges and opportunities we encountered in building this controller on Samza jobs, our experiences in deploying it on production jobs where we observed approximately 45% resource savings, and our future plans.

  • Bio: Ziting Shen is a senior software engineer in the stream processing Samza team. She has been with LinkedIn for 3.5 years and mainly focuses on monitoring, autosizing and analytics of stream processing jobs.
  • Bio: Rayman is a Staff SWE on the Samza team and has been at LI for 3 years. His interests are broadly in different aspects of distributed systems from resource efficiency, fault tolerance, reliability, performance, extensibility, and operability. Being with the team he has worked on enabling stateful Samza jobs, hot-standbys for Samza jobs, Samza autosizing, analytics, Rubik, and a few other projects.

Want to talk at a future meetup?
Please contact us via the “Contact” button in meetup.com.

The Streams team is hiring!

Related topics

Apache Kafka
Stream Processing

You may also like