Scalding the Crunchy Pig for Cascading into the Hive: Popular MapReduce tools

  • October 9, 2013 · 7:00 PM
  • Spotify NYC

Title: "Scalding the Crunchy Pig for Cascading into the Hive": Evaluating the pros and cons of popular Hadoop processing tools and frameworks.

Speaker: David Whiting, Data Engineer at Spotify

Abstract: Cascading, Scalding, Cascalog, Crunch, Scrunch, Pig, Hive - there's a plethora of options when it comes to processing your data in Hadoop, and there's always somebody with a strong opinion about which one is best for each occasion. It's often hard to get a sense of how they differ from each other and how they are good or bad for your specific use case. We will be exploring the features - both good and bad - of some of the more popular ones and showing examples of jobs implemented in each. Hopefully you'll leave with a much better idea of the philosophy behind each system and how and where you can use them.

Bio: David spent 18 months in the data team at Last.fm and since Feburary has been developing data infrastructure at Spotify - making him something of an expert in working with music data sets. He mostly works with Hadoop, but can occasionally be found dabbling in data warehousing, SQL query optimisation and front-end web apps; as well as telling everybody else they're not doing enough testing and that everything is better with static typing.

As well as generating music data, he also generates music under the guise of Demoscene Time Machine (http://music.demoscenetimemachine.com/ ), takes part in the occasional triathlon and has some very unusual dance moves.

Agenda:

• 7:00 - 7:20: Pizza & Networking

• 7:20 - 7:30: Livestream starts, intro

• 7:30 - 8:30: Dave's Talk

• 8:30 - 9:00: Q&A/Networking/Beers

Additional information:

We will be livestreaming this event. URL: http://www.livestream.com/spotifyevents (The password prompt will be disabled around 7:20)

Pizza and beverages will be available for the participants during the meetup.


Join or login to comment.

  • David W.

    Thanks for coming and for all your great questions and feedback everybody. If you missed something, the slides are available at http://thewit.ch/scalding_crunchy_pig/ and the recording should be up soon.

    2 · October 9, 2013

    • A former member
      A former member

      I had to hit Page Down to get to next slide

      October 10, 2013

    • Joshua B.

      Works fine for me on windows and mac

      October 10, 2013

  • A former member
    A former member

    As per my comment after the meeting, This meeting was extremely informative and the examples and Dave's commentary helped position me to prioritize how and why to migrate from less efficient tools analysts are using. Would like to see similar talk on Qlikview, etc. Fantastic Job. Larry

    1 · October 10, 2013

  • Greg M.

    Very informative presentation. Your perspectives on the merits and pitfalls of all the approaches discussed was very insightful. Thank you to Spotify for use of that gorgeous facility. Great beer selection and playlist. Keep up the good work at Spotify.

    October 10, 2013

  • Nitin k.

    Excellent presentation by Dave. Very good comparative analysis of various technologies with one single example. Thank you Spotify for hosting this event and for the goodies/pizzas/drinks.
    Is there a way this presentation can be shared or the files put up on meetup for benefit of everyone?

    1 · October 9, 2013

  • A former member
    A former member

    Thank you again for host Spotify offering LiveStream as a resource. It was excellent quality and available without any hitches. Dave did an excellent job, outline the tool styles, benefits/glitches and is doing so did a great job of balancing between reality of setting standards for performance and operations, as well as raising the challenge of migrating python users efforts to one of the above new approaches for better efficiency. Thanks for the HADOOP-NYC team for setting this up and SPOTIFY for giving background on the Amazon and self-managed cluster approaches. Looking forward to next talk now that you will have the capacity problem solved shortly. Larry T.

    1 · October 9, 2013

  • A former member
    A former member

    Sorry. I know link, but didn't know if password is built in or need to know.

    October 9, 2013

    • A former member
      A former member

      Thanks -- great video. See DavW's MacBook Pro (355) in screensaver mode.

      1 · October 9, 2013

    • A former member
      A former member

      Music is drowning out speaker...

      October 9, 2013

  • John

    Will it b recorded for later viewing

    October 9, 2013

    • Josh B.

      Yep, we'll post the link when it's available

      October 9, 2013

  • Josh B.

    I updated the description with an agenda(tentative), see you soon.

    1 · October 9, 2013

  • A former member
    A former member

    If someone provides live stream details I can give up on-site spot .

    October 9, 2013

  • Joshua B.

    So excited for this event today! :)

    October 9, 2013

  • Josh B.

    The event's a day away and we're looking forward to it. Since there are people on the waitlist, please update your reservation to "No" if you can't make it in person. The event will be livestreamed if you can't make it, details above.

    October 8, 2013

    • Josh B.

      From the description: The password prompt will be disabled 10-15 minutes before the speaker begins

      October 9, 2013

  • krishna b.

    krishna

    October 8, 2013

  • james

    What is the password for the livestream event?

    October 8, 2013

    • Dave C.

      "The password prompt will be disabled 10-15 minutes before the speaker begins"

      October 8, 2013

  • dhiraj k.

    Hi

    I am on waitlist . Can i still walkin to the event?

    Thanks
    Dhiraj

    1 · October 8, 2013

  • Carlos G.

    I'm Carlos.

    October 3, 2013

  • Josh Baer changed the location for this Meetup

    September 24, 2013

Our Sponsors

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more
Allison

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy