Self-service data platforms with Spark, Kafka and Avro, with Gianluca from Letgo

We hope you fully recharged your batteries this holidays, because we are starting the season with new Meetups! :)

This time we're excited to have Gianluca Amori from Letgo. He will tell us how they are redesigning their AWS data platform.

We want to thank him, and Letgo to provide us with the venue.
See you in September at Palau de Mar! Don't miss it!

---------

Title:
How to create a self-service data platform with guarantees by leveraging Avro schemas

Abstract:
Letgo is a second-hand marketplace app for buying and selling used goods locally. Based in Barcelona, with over 100M downloads and hundreds of millions of listings to date, it’s one of the most used apps in the sector in the United States.

The Letgo Data team will talk about how they are redesigning the data ingestion and data lake platforms towards AWS.

The new data lake architecture is based on a tiered design centered around the Apache Kafka ecosystem - with special mention to Kafka Connect - for data ingestion and Spark for processing and pumping data to the data lake and to anonymized data marts. The platform is built on the principles of self-servicing, compliance to data privacy laws, good development guarantees, minimal maintenance and cost containment by design.

Furthermore, we’ll describe how at the heart of our platform we define Avro schemas as a single source of truth for all our events and entities. We’ll explain how data serialization with Avro opens up wonderful possibilities like:

• tagging private fields for sensitive data and CCPA/GDPR compliance
• ensuring quality and structure of the data landing in the data lake
• transportation and consumption of data efficiently and reliably inside the platform
• self-documentation of the events

Bio:
Gianluca Amori is a Big Data Engineer at Letgo. Originally from Rome, he previously worked on software and Big Data projects for companies such as IBM, Zalando and Trovit across Italy, Ireland and Spain. He holds a master’s degree in Computer Engineering from Sapienza University of Rome.