addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1linklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

Developer Meetup: Intro to Spark Internals

This will be the first of a series of smaller meetings for people interested in *developing* Spark, that is, contributing to the codebase itself (or at least learning in depth about how it works). We'll dive into the main components of the system and the various interfaces, scheduling algorithms, etc. The goal is to give you a better understanding of Spark, and put you in a position where you could start writing your own input formats, transformation operators, or layers on top of Spark. So, overall this will not be a meeting for new users, but rather for people interested in understanding the engine.

At this first developer meetup, we'll do an introduction to the Spark codebase, including the main components, the scheduler, and the life of a query. Later next year, we'll have a similar meetup on Shark.

Yahoo! graciously offered to host the meetup, and will provide dinner. However, because the receptionists will be off-duty, you'll need to register in advance with your real name and bring some form of ID so that we can give security a list of attendees.

If you don't like to provide your real name on, email it to [masked] (but still sign up online with a fake name so we can get a count of attendees).

Doors will open at 6:30, with presentations starting at 7.

Join or login to comment.

  • Stoney V.

    A video of the Spark Internals that includes presentation slides is here:

    3 · December 21, 2012

    • Patrick W.

      This video is awesome! Thanks for adding this.

      December 21, 2012

  • Kalpit S.

    Coverage of Internals was very good. I am glad I got a spot. Looking forward to more of such talks.

    December 20, 2012

  • Matei Z.

    (By the way, the slides also contain a few more details that I didn't get to cover, and more pointers to the code.)

    December 19, 2012

  • Matei Z.

    Thanks everyone for coming! I hope this was useful. I've now uploaded the slides at We're also going to post a video.

    1 · December 19, 2012

  • Stoney V.

    Thanks so much for covering some of the internals of Spark. I am eagerly awaiting internals of the shark and then spark streaming.

    December 18, 2012

  • Nikunj M.

    Matei was crisp and very clear in explaining his material. Also felt like there was a great crowd with useful questions for Matei. Also, great to see so many of the AmpLab leads.

    December 18, 2012

  • Reynold X.

    This is getting too popular! Thanks you all for replying.

    Note that this is a meetup for developers (not users) that are considering contributing to Spark/Shark. If you won't be able to make it, we'd appreciate you giving up the spot on

    If you are serious about contributing and can't get a spot, please email Matei or me ([masked]).

    December 4, 2012

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy