Skip to content

Batch Data Processing at Spotify with Luigi

Photo of Pitt Fagan
Hosted By
Pitt F.
Batch Data Processing at Spotify with Luigi

Details

Hello everyone,

Luigi is an open-source Python framework that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows. Meetup member and software engineer Andy Sloane from Spotify is giving the talk.

Luigi also comes with Hadoop support built in (and that’s where really where its strength becomes clear). Luigi provides an infrastructure that powers several Spotify features including recommendations, top lists, A/B test analysis, external reports, internal dashboards, and many more. Here is the repo link: https://github.com/spotify/luigi
Hope to see everyone there!

Cheers,
Pitt

Photo of Big Data Madison group
Big Data Madison
See more events