A story about the refinement of a data lake with thousands of partitions

Hello everyone!

We're excited to announce our second physical meetup! This meetup will be held at the offices of Orfium, at the amphitheater on the 2nd floor, we will also stream the meetup on Zoom with a link that will be sent to all attendees the same day.

Evangelos is a Chapter Leader for Data Engineering in Moro Tech in Greece for the past 2 years while prior to that he has worked as a Big Data Engineer at BEAT. Evangelos has extensive experience with data lake architectures, scalable data pipelines, software design, and Big Data processing frameworks.

About the presentation
Join the session to learn about crucial caveats and best practices when working with a data lake with thousands of partitions built with AWS. We will take you through some of the less-known aspects of AWS Glue Crawlers and Amazon Athena that become crucial when the size of the data catalog grows. Based on this, we will discuss the problems that we initially faced, the insights we discovered along the way, and the applied solutions to achieve optimum performance without disruptions.

Meetup Outline
5' - Introduction
45' - Presentation by Evangelos Evangelou
15' - Q&A
60' - Networking/Pizzas @ the rooftop

The presentation will also be streamed in a zoom call.

We're looking forward to meeting everyone in person!

Join our Slack channel to ask any questions and keep in touch: https://awsusergroupgreece.herokuapp.com/

The event is supported by ORFIUM. ORFIUM is a leading technology company providing software, data, and licensing solutions for the entertainment industry's most complex problems around music, content, and rights management.