Name: Kubernetes + Operator + PaaSTA = Flink@Yelp
Start: 2019-09-23T18:30:00+01:00
End: 2019-09-23T20:30:00+01:00
Location: 12-16 Clerkenwell Rd

Hi everyone,

After the summer break the Flink community is back with another meetup; this time from Yelp.

The talk is scheduled to be presented at FF2019 and the team at Yelp wanted to give us a full preview! On the day we will be giving away a free ticket to attend FF2019 in Berlin, so make sure not to miss this one.

Details below. Looking forward to seeing you there!

At Yelp we run hundreds of Flink jobs to power a wide range of applications: push notifications, data replication, ETL, sessionizing and more. Routine operations like deploys, restart, and savepointing for so many jobs would take quite a bit of developers’ time
without the right degree of automation. The latest addition to our toolshed is a Kubernetes operator managing the deployment and the lifetime of Flink clusters on PaaSTA, Yelp’s Platform As A Service.

We replaced our deployment framework launching Flink clusters on top of AWS EMR with a Kubernetes operator managing fully Docker-ized Flink clusters. Compared to EMR, this architecture allowed us to both drastically reduce the deployment time of our Flink clusters
and to share our hardware resources more efficiently. In addition, we now offer to our developers the same interface they are used to for running REST services, batch jobs and many other workloads on PaaSTA.

This talk will give a brief overview of Yelp’s PaaSTA before diving into the details of how the Kubernetes operator has been implemented and how it has been integrated with Yelp developers’ workflow (deploy, logs, savepoints, upgrades, etc), to end with a glimpse
of the future features we are planning for the operator (Flink as a library, autoscaling, etc.).

Christos Hadjinikolis

Antonio

Apache Flink London Meetup

ApacheFlink

Keeping blood sugar levels up with pizzas and refreshments!

Data Reply

Technology

Hadoop

Predictive Analytics

Machine Learning

Big Data

Apache Flink

Stream Processing

Apache Spark

Data Analytics

Apache Kafka

Scala

Data Science

Database Engineers

Kubernetes + Operator + PaaSTA = Flink@Yelp

12-16 Clerkenwell Rd

Share

Apache Flink London Meetup

Kubernetes + Operator + PaaSTA = Flink@Yelp

Apache Flink London Meetup

Details

Sponsors

Members are also interested in