Using chaos engineering to level up Apache Kafka skills

Details:
18:00 - 18:30 - Mingling

18:30 - 19:15 - Using chaos engineering to level up Apache Kafka skills - Shlomi Hassan & Yaniv Ranen - Principal engineers @ ZipRecruiter

19:20 - 20:00 - Your data is in Kafka. Now what? (Hint: The answer is a data lake) - Yoni Iny - CTO @ Upsolver

--------------------------------------------------------------------------------------------------

Title:
Using chaos engineering to level up Apache Kafka skills

Abstract:
At ZipRecruiter we connect employers with jobseekers. To accomplish that, we record and process, millions of events every day using Apache Kafka. Kafka's availability is critical for our business. If Kafka is down - we are losing money.

In this talk we aim to explain the path we’ve gone through after a suffering from a production incident that occurred during a rolling upgrade. This Incident lead us to perform chaos engineering to get our team more acquainted with Kafka’s internals and how to deal with incidents.
We will share some best practices of chaos engineering Kafka and learn how to overcome failures when they appear. We will also share some of our strategies on how to perform a cluster update - Rolling upgrade vs. A blue green vs Active passive approach including pros and cons for each method.

Speakers Bio:
Shlomi Hassan - Principal engineer @ ZipRecruiter, Shlomi is a veteran working with big data systems and especially Kafka which he is using since 2012. In his last job in Kenshoo he helped create a Kafka CDC cluster replicating db's into a centric Data Warehouse using Debezium's connector.

Yaniv Ranen - Principal engineer @ ZipRecruiter, Yaniv is new to Kafka but not new to data pipelines and ETL, as he has been working for the last 20 years building them in many organizations. In his last job at Convertro (acquired by AOL) he was a part of the big data pipeline team and the data science team.

--------------------------------------------------------------------------------------------------
Title:
Your data is in Kafka. Now what? (Hint: The answer is a data lake)

Abstract:
Kafka is an amazing tool that allows organizations to collect massive amounts of data with little engineering effort.
Once the data arrives in Kafka, it needs to get offloaded. Consumers battle over resources, retention is an issue, and combining real time and offline workloads is challenging.
In this talk, I'll discuss how we solve these issues with Upsolver by building a managed data lake. Topics include:
What is a good data lake architecture?
How long should I keep my data in Kafka?
What should be a Kafka consumer and what should be a data lake consumer?
How do I build a resilient real-time app over Kafka and a data lake?

Speaker Bio:
Yoni Iny is a technologist specializing in big data and predictive analytics algorithms. He’s a co-founder and CTO at Upsolver, a Big Data startup. Before founding Upsolver, he performed several technology roles including CTO of a data science department at IDF’s elite technology intelligence unit.