Network Design Considerations and Challenges for Hadoop 'Big Data' Environments

Business decision processes have increasingly relied on frameworks that involve managing large data sets of unprecedented volume for business analytics, such as Hadoop and other distributed computing platforms.  Hadoop and the related Hadoop Distributed File System (HDFS) has been the popularly adopted open source framework of choice for mining business intelligence for its ability to process massively intensive data workloads over clusters of commodity server hardware and storage nodes.  The power of this distributed architecture, however, is directly proportional to the network infrastructure that these nodes communicate over, syncing with CPU and I/O performance. Organizations therefore have realized that the key to unlocking highly effective, optimally performing Hadoop investments is a network designed around massive scale, efficient operational manageability, and stability.  

Join us as Arista Networks, designers and implementors of some of the largest, most mission critical Hadoop clusters in the world, takes you under the covers of the network behind Hadoop with a hearty discussion on best practice network architectures for these demanding workloads and the power of EOS - Arista’s programmable Extensible Operating System.  We’ll dive deep into EOS and its uniquely integrated feature sets specifically focused on Hadoop/HDFS including MapReduce Tracer, RAIL, and Zero Touch Provisioning (ZTP).  Get answers to questions such as “how much visibility do we have into these Hadoop workloads?”, “how much data is each job sending over the network?”, “what are the current map and reduce tasks?”, and “how can I find and troubleshoot a worrisome node behind a particular interface?”

Guest speakers will include Benoit Sigoure of Arista Networks.  Benoit is the original author of OpenTSDB, the distributed monitoring system built on top of HBase. He also wrote AsyncHBase, the alternative HBase client that is fully asynchronous, non-blocking, and thread-safe. Benoit is currently working on building distributed systems for next-generation datacenter networks at Arista Networks. He also works on network extensibility, Hadoop integration, APIs, and network programmability in general.

Join or login to comment.

  • Mike P.

    I recorded the meetup using my Google Glass. Unfortunately, Glass only has about 45 minutes of battery life for recording video, so I skipped most of the Q&A. The videos are syncing to my Google Drive now: https://drive.google.com/folderview?id=0B0B2VcpkcY6wTm5pUXVCTGYwV3c&usp=sharing

    June 26

  • Robert Del R.

    I am working with Jeremiah (Chief Creative Officer) and Oliver (CTO) to turn my article: “Finding Order Amid Chaos: Big Data Analytics, the 2008 Financial Crisis and Science Fiction Becoming Fact” and others into a Blog, which we call the JOBBlog, after our first initials. I am Bob and I put the “B” in JOB and I am the Publisher, since all the content will come from me!

    The “Big Data” article is 93 pages and growing, so it will become a book! (My first book). I feel it is “original” in that it places Big Data in historical perspective, by comparing it to its prior expression in Science Fiction.

    June 23

Our Sponsors

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more
Bill

I started the group because there wasn't any other type of group like this. I've met some great folks in the group who have become close friends and have also met some amazing business owners.

Bill, started New York City Gay Craft Beer Lovers

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy