SF: Next Generation Hadoop Architecture with Roman Shaposhnik


The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture with Roman Shaposhnik

6:00-6:30pm Pizza and Networking
6:30-8:00pm Talk and Q&A
8:00-8:30pm Wind down

The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture
with Apache Hadoop contributor, Roman Shaposhnik

In this talk, I will go through the evolution of Hadoop and its ecosystem projects and will try to peer into the crystal ball to predict what may be coming down the pike. I will discuss various way of crunching the data on Hadoop (MapReduce, OpenMPI, Spark and various SQL engines) and how these tools compliment each other.

Apache Hadoop is no longer just a faithful, open source, scalable implementation of two seminal papers that came out of Google 10 years ago. It has evolved into a project that provides the enterprises with a reliable layer for storing massive amounts of unstructured data (HDFS) while allowing different computational frameworks to leverage those datasets.

The original computational framework (MapReduce) has evolved into a much more scalable set of general purpose cluster management APIs collectively known as YARN. With YARN underneath, MapReduce is still there to support batch-oriented computations, but it is no longer the only game in town. With OpenMPI, Spark, and Tez rapidly becoming available now is truly the most exciting time to be a developer in a Hadoop ecosystem. It is also the time when you don't have to be employed by Yahoo!, Facebook or EBay to have access to mind-blowing compute power. That power is a credit card and a pivotal.io account away from anybody on the planet.

I will conclude by outlining some of the ongoing work that makes Hadoop and its ecosystem projects first class citizens in cloud environments based on the work that Pivotal engineers have done with integrating Hadoop into PivotalONE PaaS.

Bio: Roman Shaposhnik is a Sr. Manager Manager of Hadoop Open Source Platform at Pivotal Inc. He is a committer on Apache Hadoop, and holds a Chair position in Apache Bigtop and Apache Incubator projects. Roman has been involved in Open Source software for more than a decade and has hacked projects ranging from Linux kernel to the flagship multimedia library known as FFmpeg. He grew up in Sun microsystems where he had an opportunity to learn from the best software engineers in the industry. Roman's alma mater is St. Petersburg State University, Russia where he studied to be a mathematician.

Join or login to comment.

  • Trip S.

    can we pls have the slides posted

    May 1, 2014

  • Roman V S.

    Thanks for the kind words! Really appreciated! If you're interested in more hands-on meetups, keep an eye on the ones we're going to run in Palo Alto. Today was the zeroth inaugural one on Apache YARN. More hands-on goodness to come (checkout my twitter for updates @rhatr).

    1 · April 30, 2014

  • Vman M.

    Great investment of time... didn't feel bad missing all the 40+ other meetups on Hadoop/BD

    April 30, 2014

  • Andrew H.

    Spectacularly good overview of Hadoop ecosystem. Well presented take-aways and good analogies. Thanks

    1 · April 30, 2014

  • Tom L.

    great talk

    1 · April 30, 2014

  • Philip Shanks, CISSP, C.

    Great presentation, thanks Roman! And thanks to Pivotal for hosting!

    1 · April 30, 2014

  • Wayne L.

    Great flow and one of the better hadoop overviews I've heard

    2 · April 29, 2014

  • Vamsi M.

    Very good presentation!

    2 · April 29, 2014

  • Chester C.

    The title is misleading. The talk is not about next generation Hadoop Architecture. The title should be "Hadoop ecosystem Overview"

    1 · April 29, 2014

  • A former member
    A former member

    Presentation was great but need more pizzas :)

    1 · April 29, 2014

  • Suhas

    Will the event be recorded?

    1 · April 29, 2014

    • Tamao N.

      Yes, we try to record all of our meetups and they will be released mostly during the summer when the meetup group takes a break.

      April 29, 2014

  • Guolin C.

    Sorry, can not make it today.

    April 29, 2014

  • Guolin C.


    April 22, 2014

Our Sponsors

  • Pivotal

    Pivotal offers venues, food, beverages, snacks, and ping pong!

People in this
Meetup are also in:

Sometimes the best Meetup Group is the one you start

Get started Learn more

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy