Developer Meetup: Intro to Spark Internals

This will be the first of a series of smaller meetings for people interested in developing Spark, that is, contributing to the codebase itself (or at least learning in depth about how it works). We'll dive into the main components of the system and the various interfaces, scheduling algorithms, etc. The goal is to give you a better understanding of Spark, and put you in a position where you could start writing your own input formats, transformation operators, or layers on top of Spark. So, overall this will not be a meeting for new users, but rather for people interested in understanding the engine.

At this first developer meetup, we'll do an introduction to the Spark codebase, including the main components, the scheduler, and the life of a query. Later next year, we'll have a similar meetup on Shark.

Yahoo! graciously offered to host the meetup, and will provide dinner. However, because the receptionists will be off-duty, you'll need to register in advance with your real name and bring some form of ID so that we can give security a list of attendees.

If you don't like to provide your real name on meetup.com, email it to matei@eecs.berkeley.edu (but still sign up online with a fake name so we can get a count of attendees).

Doors will open at 6:30, with presentations starting at 7.