addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwchatcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrosseditemptyheartexportfacebookfolderfullheartglobegmailgoogleimageimagesinstagramlinklocation-pinmagnifying-glassmailminusmoremuplabelShape 3 + Rectangle 1outlookpersonplusprice-ribbonImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruseryahoo

Hadoop Ops

Pizza, beer, and mingling.

Lightning Talks:

  • "Tracking multi-tenant resource usage with 'White Elephant'"
    Adam Faris - Senior Grid Operations System Engineer, LinkedIn

"Managing and Provisioning Hadoop Clusters"
Darren Lo - Software Engineer, Cloudera

"How Klout migrated from CDH3 to CDH4"
Ian Kallen - Lead Ops Engineer, Klout

"Challenges and Capabilities in Managing a MapR Cluster"
David Tucker - Solutions Architect, MapR Technologies

"Hadoop Cluster Management with Apache Ambari"
Sumit Mohanty - Member of Technical Staff @Hortonworks & Apache Ambari PMC Member/Committer

There's certainy a lot of interest in using Hadoop, as there are meetups, classes, and tutorials on writing Map-Reduce jobs. But there isn't too much discussion on how to manage Hadoop clusters.

If you're using Hadoop in production, how do you manage it? Does the distribution you're using provide any tools to make the job easier? What are the pitfalls? Are there parts of the system that are less robust or that have problems more often? Are you running Hadoop on bare metal, or in a cloud environment, and is one easier than the other? Are you using a managed solution such as AWS' Elastic Map Reduce, and if so, what does that bring to the table as far as operations goes?

For our September event, I'd like to hear about issues and solutions in managing a Hadoop deployment. I'm looking for[masked] minute talks. If you can give a talk, please contact me, Chris Westin, through meetup.

As well as the evening's theme talks, we can fit in 2-3 five minute lightning talks at the beginning of the evening; any topic that would be interesting to the #lspe audience is welcome. If you're interested in giving a lightning talk, contact me, Chris Westin, through meetup.

Join or login to comment.

  • Arul R.


    I would like to get the presentation slides for yesterday's meetup. Can you please help.


    September 20, 2013

    • Chris W.

      I've asked the presenters for their slides, and will post them here when I've gotten them all. It usually takes a few days for them to respond.

      September 20, 2013

  • Alec C.

    Nice to see all camps represented.
    Also because of the post presentation questioning I wonder how many people are actually using dynamically scaled clusters. We certainly don't. Occasionally we'll spin up another cluster to catch up a job but the regular one simply grows as we add data.

    Any chance of getting the slides?

    2 · September 20, 2013

    • Chris W.

      As always, I've sent a message to the speakers asking for their slides. When I've gotten all the ones they're willing to share, I'll post them here. It usually takes a few days to get them all.

      1 · September 20, 2013

    • Allen W.

      The bigger problem with "autoscale" is that it isn't that compatible with using the (dfs|mapred}.hosts params that allow for decommissioning of services (pretty much required at large scales) or for Hadoop-level security without a lot of trickery/glue.

      1 · September 20, 2013

  • Myra H.

    Great to see Cloudera, MapR and HortonWorks there, and all presentations are very informative.

    1 · September 19, 2013

  • Arul R.


    1 · September 19, 2013

  • Murali

    More product centric especially from Cloudera.

    September 19, 2013

  • Chris W.

    Folks, we've got 2 short talks, and 3 normal talks (20-25 mins), so I want to get started early this time, at 6:15PM. I'm sure you all know Bay Area traffic is pretty bad at this time, so please give yourself enough time to get to URL's on time to avoid missing the first talk.

    September 14, 2013

Our Sponsors

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy