[SF]Deep Dive: Spark Project Tungsten: Largest Performance Optimizations to Date

Deep dive into the CPU and Memory optimizations implemented as part of Spark 1.5's Project Tungsten.

We'll focus on JVM code generation, CPU-cache locality, self-managed (Unsafe) garbage collection, and key performance tuning recommendations.

Agenda

6:30-7pm: Arrive and Mingle

7-7:15pm: Announcements and Updates

7:15-7:30pm: Highlights of the Spark Summit Europe 2015 (Amsterdam)

7:30-8:30pm: Code-level, deep dive into the performance and memory optimizations of Project Tungsten as part of Spark 1.5.

Relevant Links:

https://databricks.com/blog/2015/09/09/announcing-spark-1-5.html (http://www.slideshare.net/SparkSummit/deep-dive-into-project-tungsten-josh-rosen)

http://www.slideshare.net/SparkSummit/deep-dive-into-project-tungsten-josh-rosen

https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html

https://issues.apache.org/jira/browse/SPARK-8159

https://issues.apache.org/jira/browse/SPARK-7080

http://0x0fff.com/spark-architecture-shuffle/

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala