NOTE: THIS MEETUP IS IN SAN FRANCISCO at 5:45pm PDT
The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture with Roman Shaposhnik
6:00-6:30pm Pizza and Networking
6:30-8:00pm Talk and Q&A
8:00-8:30pm Wind down
The Elephant in the Cloud: A Quest for the Next Generation Hadoop Architecture
with Apache Hadoop contributor, Roman Shaposhnik
In this talk, I will go through the evolution of Hadoop and its ecosystem projects and will try to peer into the crystal ball to predict what may be coming down the pike. I will discuss various way of crunching the data on Hadoop (MapReduce, OpenMPI, Spark and various SQL engines) and how these tools compliment each other.
Apache Hadoop is no longer just a faithful, open source, scalable implementation of two seminal papers that came out of Google 10 years ago. It has evolved into a project that provides the enterprises with a reliable layer for storing massive amounts of unstructured data (HDFS) while allowing different computational frameworks to leverage those datasets.
The original computational framework (MapReduce) has evolved into a much more scalable set of general purpose cluster management APIs collectively known as YARN. With YARN underneath, MapReduce is still there to support batch-oriented computations, but it is no longer the only game in town. With OpenMPI, Spark, and Tez rapidly becoming available now is truly the most exciting time to be a developer in a Hadoop ecosystem. It is also the time when you don't have to be employed by Yahoo!, Facebook or EBay to have access to mind-blowing compute power. That power is a credit card and a pivotal.io account away from anybody on the planet.
I will conclude by outlining some of the ongoing work that makes Hadoop and its ecosystem projects first class citizens in cloud environments based on the work that Pivotal engineers have done with integrating Hadoop into PivotalONE PaaS.
Bio: Roman Shaposhnik is a Sr. Manager Manager of Hadoop Open Source Platform at Pivotal Inc. He is a committer on Apache Hadoop, and holds a Chair position in Apache Bigtop and Apache Incubator projects. Roman has been involved in Open Source software for more than a decade and has hacked projects ranging from Linux kernel to the flagship multimedia library known as FFmpeg. He grew up in Sun microsystems where he had an opportunity to learn from the best software engineers in the industry. Roman's alma mater is St. Petersburg State University, Russia where he studied to be a mathematician.