Past Meetup

Developer Meetup: Intro to Spark Internals

This Meetup is past

64 people went

Location image of event venue


This will be the first of a series of smaller meetings for people interested in *developing* Spark, that is, contributing to the codebase itself (or at least learning in depth about how it works). We'll dive into the main components of the system and the various interfaces, scheduling algorithms, etc. The goal is to give you a better understanding of Spark, and put you in a position where you could start writing your own input formats, transformation operators, or layers on top of Spark. So, overall this will not be a meeting for new users, but rather for people interested in understanding the engine.

At this first developer meetup, we'll do an introduction to the Spark codebase, including the main components, the scheduler, and the life of a query. Later next year, we'll have a similar meetup on Shark.

Yahoo! graciously offered to host the meetup, and will provide dinner. However, because the receptionists will be off-duty, you'll need to register in advance with your real name and bring some form of ID so that we can give security a list of attendees.

If you don't like to provide your real name on, email it to [masked] (but still sign up online with a fake name so we can get a count of attendees).

Doors will open at 6:30, with presentations starting at 7.