November 2019 Bay Area Apache Flink Meetup @ Cloudera

Excited to announce the next Bay Area Apache Flink meetup on November 20 hosted by Cloudera! Come hear the talks by Gyula Fora (Cloudera) and Lakshmi Rao (Lyft) and network with fellow members of the Bay Area Flink community in a relaxed atmosphere.

Agenda:

6:30 pm - Kick-off & networking
7:00 pm - 7:30 pm - Talk #1: Log aggregation for Flink pipelines by Gyula Fora, Cloudera
7:30 pm - 8:00 pm - Talk #2: Running Flink in production: The good, the bad and the in-between by Lakshmi Rao, Lyft
8:00 pm - 8:30 pm - Networking

----------------------------------------------------------------------------------------------------

Talk #1: Log aggregation for Flink pipelines

Abstract: Log aggregation and monitoring is a common challenge in data processing applications. We will show you how you can build your own customizable
logging solution based on components that are readily available in the Cloudera platform.
We will cover the following components:

• Configuring Kafka based logging for Flink jobs
• Implement scalable real-time log indexing in Flink
• Log search and dashboards using Solr and Hue

Speaker: Gyula Fora, Software Engineer at Cloudera
Bio: Gyula is a Software Engineer in the Flink Engineering team at Cloudera working on integrating Flink into the Cloudera platform.
He has been a committer and contributor since the early days of Flink streaming and has used Flink in large scale production at King for almost 4 years delivering innovative real-time applications at a global scale.
Gyula grew up in Budapest where he first started working on distributed stream processing and later became a core contributor to the Apache Flink project. Gyula has been a speaker at numerous big data related conferences and meetups, talking about stream processing technologies and use-cases.

Talk #2: Running Flink in production: The good, the bad and the in-between

Abstract: The streaming platform team at Lyft has been running Flink jobs in production for more than a year now, powering critical use cases like improving pickup ETA accuracy, dynamic pricing, generating machine learning features for fraud detection, real-time analytics among many others. Broadly, the jobs fall into two abstraction layers: applications (Flink jobs that run on the native platform) and analytics (that leverage Dryft, Lyft’s fully managed data processing engine). This talk will give an overview of the platform architecture, deployment model and user experience. The talk will also dive deeper into some of the challenges and the lessons that were learnt, running Flink jobs at scale, specifically around scaling Flink connectors, dealing with event time skew (source synchronization) and highlight common patterns of problems observed across several Flink jobs. Finally, the talk will give insights into how we are re-architecting the streaming platform @ Lyft using a Kubernetes based deployment.

Speaker: Lakshmi Rao, Software Engineer at Lyft
Bio: Lakshmi is a software engineer on the streaming platform team at Lyft. The team builds and supports the core infrastructure that enables several product teams at Lyft to easily and reliably spin up Flink jobs to perform aggregations on real-time data. Most recently, she has been spending time re-architecting the platform to a Kubernetes based deployment. Prior to Lyft, Lakshmi worked in fin-tech land, building a search and information retrieval platform for Goldman Sachs.

----------------------------------------------------------------------------------------------------

Share the meetup on Twitter and invite your friends & colleagues to join!