NYC: Next Generation Hadoop Architecture - talk and hands-on Pivotal Hadoop

  • March 20, 2014 · 6:00 PM

NOTE: THIS MEETUP IS IN NEW YORK CITY at 6:00pm EST

The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture - talk and hands-on Pivotal Hadoop
March 20, 2014 - Pivotal Labs office NYC
6:00-7:00 pm EST gyros and other food and networking!
7:00-8:00 pm EST The Elephant in the Cloud with Roman Shaposhnik
8:00-9:00 pm EST Hands-on Pivotal Hadoop

(Please note all times are in EST even if the settings on this meetup page are not).

The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture
with Apache Hadoop contributor, Roman Shaposhnik (7:00-8:00 pm)

In this talk, I will go through the evolution of Hadoop and its ecosystem projects and will try to peer into the crystal ball to predict what may be coming down the pike. I will discuss various way of crunching the data on Hadoop (MapReduce, OpenMPI, Spark and various SQL engines) and how these tools compliment each other.

Apache Hadoop is no longer just a faithful, open source, scalable implementation of two seminal papers that came out of Google 10 years ago. It has evolved into a project that provides the enterprises with a reliable layer for storing massive amounts of unstructured data (HDFS) while allowing different computational frameworks to leverage those datasets.

The original computational framework (MapReduce) has evolved into a much more scalable set of general purpose cluster management APIs collectively known as YARN. With YARN underneath, MapReduce is still there to support batch-oriented computations, but it is no longer the only game in town. With OpenMPI, Spark, and Tez rapidly becoming available now is truly the most exciting time to be a developer in a Hadoop ecosystem. It is also the time when you don't have to be employed by Yahoo!, Facebook or EBay to have access to mind-blowing compute power. That power is a credit card and a pivotal.io account away from anybody on the planet.

I will conclude by outlining some of the ongoing work that makes Hadoop and its ecosystem projects first class citizens in cloud environments based on the work that Pivotal engineers have done with integrating Hadoop into PivotalONE PaaS.

Bio: Roman Shaposhnik is a Sr. Manager Manager of Hadoop Open Source Platform at Pivotal Inc. He is a committer on Apache Hadoop, and holds a Chair position in Apache Bigtop and Apache Incubator projects. Roman has been involved in Open Source software for more than a decade and has hacked projects ranging from Linux kernel to the flagship multimedia library known as FFmpeg. He grew up in Sun microsystems where he had an opportunity to learn from the best software engineers in the industry. Roman's alma mater is St. Petersburg State University, Russia where he studied to be a mathematician.

Hands-on Pivotal Hadoop (8:00-9:00 pm)
Come Test Drive The World's Most Advanced Analytical Platform on The World's Largest Public Hadoop Cluster:

We will introduce the most recent capabilities of Pivotal HD, the HAWQ SQL Query Engine, and GemfireXD in memory capabilities over HDFS. In this hands-on lab Pivotal will be providing one-time access to attendees to test drive the 1000 node Analytics Workbench cluster, the world's largest publicly accessible Hadoop platform. We will walk through sample data sets and analytical query packages and methods and then attendees will be set loose to enjoy our leading technologies on this one-of-a-kind platform.

Laptop requirements:
Network connectivity : Ability to connect to our guest wireless.

Access : Ability to perform ssh.
- On a Mac, this is built in.
- For Windows an ssh app is required. PuTTY is usually a good recommendation

Join or login to comment.

  • John Peter S.

    Very good overview by Roman. Couldn't stay for the hand's on though.
    Will there be instructions posted on how to use the 1000 node work cluster with the thumb drive, or was this one time and on-site only?
    Oh, and thanks to Pivotal for the food. Quite good!

    1 · March 20, 2014

    • Tamao N.

      Hi John - I'll check with the training team on what we can do.

      March 21, 2014

    • Tamao N.

      Sorry for the delay! Here are the instructions for the single-node PHDs that we handed out in NYC: http://pivotalhd.docs...­

      April 29, 2014

  • Trip S.

    we are still waiting for slides and the video

    1 · March 28, 2014

    • Tamao N.

      http://www.slideshare...­

      Note: if you download the pdf from slideshare it renders correctly, but if you simply browse throw their web interface all the labels on the little architectural boxes are gone. Video is on the way.

      2 · March 31, 2014

    • Trip S.

      Thanks

      April 1, 2014

  • Pete

    Yes, I would also like access to the slides and the video as well if that is available.

    1 · March 29, 2014

    • Tamao N.

      http://www.slideshare...­

      Note: if you download the pdf from slideshare it renders correctly, but if you simply browse throw their web interface all
      the labels on the little architectural boxes are gone.

      1 · March 31, 2014

    • Tamao N.

      video is on the way.

      March 31, 2014

  • Trip S.

    Cool - these are the best educational stuff

    1 · March 24, 2014

  • Trip S.

    can we pls post the slides

    1 · March 22, 2014

  • Larry T.

    Roman has the credentials to pull off this talk! He shared a lot of points of view about Apache, all the spinoff projects, importance of Google papers, then launched into Hadoop 2.0 and gave a great summary of "when" each technology became relevant. Great insight into Cloudera to Pivotal. Much appreciate the SPARK overview -- already running the Giveaway USB and the SPARK download (UCB/AMP). Mark Secrist led the labs and fit pig, hive, SQL / HAWQ together. Tamao -- thank you again -- great hospitality, intelligent banter, and know-how delivered! A perfect Meet-Up!

    1 · March 21, 2014

  • JT

    This one rocked! I wish all meetups were bread baskets like this. Kudos to the Pivotal team, 10/10.

    2 · March 21, 2014

  • Ashok K.

    Here is the link to the lab files shared by Mark: http://tinyurl.com/nyc-meetup-labs

    1 · March 21, 2014

  • Eric S.

    Really informative session! Thank you Roman for a great presentation on the evolution of Hadoop ecosystem, comparisons on the different technology offerings, and the future trend. Thank you Mark for leading a great hands-on session to demonstrate/compare Pig, Hive, and HAWQ. Last but not the least, a big THANK YOU to Tamao for organizing this very insightful event! I look forward to more events like this in the future.

    2 · March 21, 2014

  • Nitin k.

    Great work Roman for presenting an unprejudiced opinion on the various Hadoop technologies available today. Very well done slides. Thanks to Tamao for arranging this, great venue and equally good food. Kudos to Pivotal for sharing their knowledge and insights....

    1 · March 21, 2014

  • Prateek R.

    Very detailed discussion on technology followed by detailed practical labs for PIG and HIVE.

    1 · March 21, 2014

  • Ram

    Good intro to Pivotal role in Hadoop space

    1 · March 21, 2014

  • A former member
    A former member

    Very informative presentation. Thanks Roman for the detailed explanations. I would love to get a copy of the slides if possible.

    1 · March 21, 2014

  • Ashok K.

    Excellent Presentation, Fantastic Demo and Amazing food. Thanks to Roman, Mark, Tamao and Pivotal for arranging this!

    2 · March 20, 2014

  • Vman M.

    A live stream would be ideal here...

    2 · March 20, 2014

    • Vman M.

      St Petersburg, Russia, please :-)

      1 · March 20, 2014

    • Tamao N.

      :)

      March 20, 2014

  • Nitin k.

    Is this going to be recorded and the link shared...
    There are people who will benefit a lot from this....
    Also, would be great if slides can be put up...

    March 20, 2014

    • Tamao N.

      We hope to as long as the recording goes well :)

      March 20, 2014

  • Larry T.

    Any chance the laptop requirements will be updated?

    March 19, 2014

    • Tamao N.

      Hi Larry - I hope you saw the update on this page as well as the email that we sent out to registrants. Please let me know if you have further questions. Look forward to seeing everyone!

      March 20, 2014

  • Vitthal G.

    Great! Looking forward to recording for remote folks!

    2 · March 4, 2014

    • Sanjay S.

      Awesome...Thanks a lot..

      March 5, 2014

Our Sponsors

  • Pivotal

    Pivotal offers venues, food, beverages, snacks, and ping pong!

People in this
Meetup are also in:

Imagine having a community behind you

Get started Learn more
Henry

I decided to start Reno Motorcycle Riders Group because I wanted to be part of a group of people who enjoyed my passion... I was excited and nervous. Our group has grown by leaps and bounds. I never thought it would be this big.

Henry, started Reno Motorcycle Riders

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy