Batch Data Processing at Spotify with Luigi

Hello everyone,

Luigi is an open-source Python framework that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows. Meetup member and software engineer Andy Sloane from Spotify is giving the talk.

Luigi also comes with Hadoop support built in (and that’s where really where its strength becomes clear). Luigi provides an infrastructure that powers several Spotify features including recommendations, top lists, A/B test analysis, external reports, internal dashboards, and many more. Here is the repo link: https://github.com/spotify/luigi
Hope to see everyone there!

Cheers,
Pitt

Join or login to comment.

  • Andy S.

    Someone asked whether Spotify has a development blog, and I came up with the wrong link -- the right one is here: http://labs.spotify.com/

    Also @erikbern has a blog about machine learning stuff at Spotify and in general: http://erikbern.com/

    1 · August 26

  • Yash

    It was really eye opening and great experience for new graduate student like us. It will be good to have more hands on experience if possible.

    2 · August 26

  • Brad O.

    Gonna have to duck out at the last second - too much stuff going on with Forward Tech :(

    August 26

  • Andy S.

    The overture street garage is across the corner, and there is parallel parking on Mifflin street (not sure if it's metered after 6 or not)

    August 26

  • mohideen k.

    Where do i park my car?

    August 26

  • Mike C.

    Healthcare Big Data Startup

    August 24

Our Sponsors

  • Cloudera

    Cloudera is the general sponsor of Big Data Madison.

People in this
Meetup are also in:

Imagine having a community behind you

Get started Learn more
Rafaël

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy