Spark Catalyst: A Functional Query Optimization Engine

UPDATE: We are still on for tonight!!

Russel Spitzer will be speaking! And The Ace hotel will be hosting!

Spark Catalyst: A Functional Query Optimization Engine

Learn how Apache Spark uses immutable trees and pure functions to transform SQL Queries into distributed code. This is particularly interesting because while the user facing optimization engine is Functional, the system compiles the eventual trees into mutable Java based code for performance. By separating the mechanics of the implementation from the description of the computation, Spark is able to perform provably safe optimizations in a very extensible system. As an example of the extensibility, I'll show how a new custom join can be added to the default operations.