Securing Apache Spark on Production Hadoop Clusters

Venue accommodates 80 people, please RSVP only if you will attend. At this session, we will provide two 20-minute talks:

Secure Spark: An update

With Spark being used for more and more production workloads with stringent security requirements, fully locking down Spark applications has become critical. We will begin by first providing a brief overview of basic security concepts including authentication and encryption. Next we will discuss recent improvements to Spark 1.6 to add native over-the-wire and on disk encryption using SASL. We will conclude by discussing remaining challenges and work items in Spark security.

Livy: A unified REST web service for Spark

Livy is an open source Apache licensed REST web service that manages long running Spark contexts in your cluster. By utilizing Livy, clients can easily submit:1. Spark jobs programmatically using a thin client2. Spark code snippets that are compiled and run in the cluster3. Entire Spark applications as JARsLivy effectively makes it possible to build both interactive web/mobile applications and multi-tenant notebooks. In this talk we will provide a brief overview of Livy, its architecture, APIs and future work and how you can contribute!

Kostas Sakellis is a Software Engineer and contributor to Apache Spark. Previous to that, he contributed to the extensibility effort on Cloudera Manager. Before Cloudera, Kostas did a 6 year stint at Amazon working across various teams including platform infrastructure. Kostas has a Bachelors of Mathematics in Computer Science from the University of Waterloo.