Deep Dive into Spark, Tachyon, and Mesos Internals

Location Information

Address: WhitePages.com Rainier Tower
1301 – 5th Ave Ste. 1600 Seattle, WA 98101

Doors on the East (5th Ave) entrance are locked after 6pm. Please use the West (4th Ave) entrance for access to the building after 6pm. The elevators will be unlocked for floor 16 only.

Parking is available in our building and is valet only. Cost is $12.00 for 1-2 hours, $15 for 2-3 hours and $19 for 3-4 hours. $8.00 after 6pm. Enter on Union between 4th & 5th) Additional parking can be found in the Hilton Parking Garage. Cost is $6.00 after 5pm. Enter on 6th Ave between University and Union.

Agenda

• 6:15pm: Come to White Pages

• 6:30pm: Meetup Logistics discussion, Intros by Datameer

• 6:45pm: Session Starts

• 7:30pm: Q&A

with ~15min buffer

Session Specifics

This is an exciting deep dive into the Spark and Mesos Internals. Subject examples include Atigeo's open source contributions to the Apache Spark and Mesos projects:

• Spark SQL/Shark http server ( https://github.com/Atigeo/jaws-spark-sql-rest )

• Spark Job Server that resolves the Spark core’s current bug of running multiple contexts in the same JVM ( https://github.com/Atigeo/spark-job-rest )

• Updates to the current Spark framework schedulers when running on Mesos (fine-grained and coarse-grained modes)

• Tachyon: hdfs vs native APIs, RawTable

• Spark optimizations: shuffle, kryo

• Mesos Framework Starvation bug: running multiple Spark/Shark servers alongside Hadoop and Aurora on top of Mesos

There will be almost no slides in this session as Claudiu will spend all of his time in live demos and deep into the core of Spark/Shark/Tachyon/Mesos code in his favorite IDE.. This would be an advanced class for developers only, prior familiarity with Spark and Mesos is highly recommended