Bay Area Hadoop User Group (HUG) Feb 2011 Meetup

Detailed agenda and summaries to follow. General agenda:

 

  • 6:00 - 6:30 - Socialize over food and beer(s)
  • 6:30 - 7:00 - Next Generation Hadoop
  • 7:00 - 7:30 - Next Generation Hadoop Operations at Facebook

 

Next Generation Hadoop MapReduce: Talk about Apache Hadoop[masked] branch and the next generation of Hadoop MapReduce. The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization.

 

http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/

 

Presenter: Arun C Murthy, Yahoo!

 

Next Generation Hadoop Operations at Facebook: Hadoop's traditional role as a framework for batch-oriented execution of map-reduce jobs is rapidly expanding to include many other use cases, such as Hbase, Scribe,  and low latency ad-hoc queries of large datasets. Downtime is becoming less acceptable, and existing map-reduce jobs continue to get larger, with tighter expectations around completion time. Storage and data retention requirements continue to grow. Quite simply, it is both an amazing and extremely challenging time to be a Hadoop administrator.

In this talk we will discuss the challenges facing our operations team in 2011 growing and managing a variety of Hadoop clusters throughout Facebook, and the solutions we are developing to address them. We will share our key best practices and lessons learned, and how they can be applied to any organization.

Presenter: Andrew Ryan, Facebook

Yahoo Campus Map:

Detail map

Location on Wikimapia:

http://www.wikimapia.org/#lat=[masked]&lon=[masked]&z=18&l=0&m=b&search=yahoo

 

Join or login to comment.

  • Owen O.

    The slides and videos are posted at http://developer.yahoo.com/blog...­

    February 23, 2011

  • stephen b.

    Great job!

    February 20, 2011

  • A former member
    A former member

    Very relevant to me as a Hadoop Developer.

    February 18, 2011

  • Prashanth S.

    This was my first tryst with group meeting. I liked it! Plan to attend the next one too!

    February 17, 2011

  • Chi H.

    I'm new to Hadoop and the one data point that I was very impressed with is that Yahoo has maxed out their cluster(s) and have a total of 12k nodes.

    The Facebook preso was nothing more than a rundown of sysadmin practices, which can be applied to any system, so I was unimpressed.

    February 17, 2011

  • J F

    Need to ask people to go outside if they are talking at back of room. Can't hear speaker due to all the chit chat.

    February 17, 2011

  • A former member
    A former member

    Arun's talk was very interesting. I would have liked to see him enumerate the differences between what Yahoo is doing and Mesos. I also enjoyed Andrew Ryan's talk.

    February 17, 2011

  • Deepak J.

    very good

    February 17, 2011

  • Bill B.

    Thanks to the speakers for presenting how they are using it today and wants coming in the future.

    February 17, 2011

  • Cesar D.

    The presentation were great and the topics were very interesting. I do with the people in the back could have kept their conversations low. The noise from people not participating really got in the way of clearly listening to the speaker.

    February 17, 2011

  • Owen O.

    I sent out the email to all, but we are holding the HUG in building E this month. The Yahoo campus map is here: http://photos3.meetupstatic.com...­

    February 15, 2011

  • Yahoo! HUG O.

    @Ajoy: I can understand the position Van is in, sort of mirrors mine. HUG consistently meet on the 3rd Wednesday and we have our venue reservation at Yahoo! set for 2011. :-) I will give it a try, may be we can move starting Q2.

    February 10, 2011

  • Ajoy B.

    @Avik: I had exchanged messages with Van Riper, the SV JUG organizer last month when the Jan events clashed: http://www.meetup.com/sv-web-ju...­
    From what he said, SV JUG consistently meets on the 3rd Wednesday and they have their venue reservation at Google set for 2011. If we want to avoid future conflicts, this is something to consider. I would expect considerable overlap of interest in the 2 groups.

    February 10, 2011

  • Yahoo! HUG O.

    @Lee: Thanks. Unfortunately because of various logistics it will not be easy to move the HUG session at this point. If I had seen interest earlier from more folks on moving this because of the conflict it might have been possible.

    February 9, 2011

  • Lee

    @Ajoy @Avik: The SV JUG meetup (What's New in Spring 3.1) is also on the same night as this meetup.

    February 9, 2011

  • Vladimir R.

    "low latency ad-hoc queries of large data sets" in Next Generation Hadoop? Wow! We will get a Dremel!

    February 9, 2011

  • Yahoo! HUG O.

    Vijay - We agree. :-) That's why we switched to the URL's Cafe in Bldg C at the same campus, starting from this HUG. This is a larger space, so those who want to network have plenty of room to do so. This location also has better audio/video and acoustics. Do send me some feedback after you attend this time. /Thanks.

    February 9, 2011

  • Vijay B.

    Avik, Any way to boost audio there with extra speakers ?
    Was really hard to hear in the back. The chatter amongst the attendees made it worse.

    February 9, 2011

  • Yahoo! HUG O.

    @Ajoy: Will look in to it. Not sure what the attendee overlap is? That might influence a decision one way or another.

    February 3, 2011

  • Ajoy B.

    Like the Jan meetup, this is also on the 3rd Wednesday of the month. Want to note that the 3rd Wednesday is also the day for all Silicon Valley Web JUG meetups. Although they do not have one in February (at least, not on the schedule yet), the JUG & HUG events happened at the same time in Jan. So is it possible that HUG Meetups be held on a day other than the 3rd Wednesday?

    February 3, 2011

Our Sponsors

  • Yahoo! Inc.

    Meeting space, pizza and drinks are sponsored by the Yahoo! Hadoop team.

People in this
Meetup are also in:

Sometimes the best Meetup Group is the one you start

Get started Learn more
Rafaël

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy