Developer Meetup: Why understanding how Spark is built helps the community

Developer Talk: Why understanding how Spark is built helps the developer community. Detailed abstract and speaker bio below. This talk will be filmed and posted on the Apache Spark YouTube page.

Agenda:

6:30 - 7:00 :: Mingling

7:00 - 7:05 :: Welcome

7:05 - 8:15 :: Technical talk

8:15 - 9:00 :: Mingling

To enter the building: At security – let them know you are there for the Spark Meetup and need to go to Capital One – 5th Floor. All guests must register once on the 5th floor.

Abstract: Ever wonder what goes into building, testing, and maintaining a codebase such as Spark? Building a system that others can contribute to pain-free out of the box is a daunting task, but one that, when done right, engages the community and fosters long term growth. In this talk we'll discuss how Spark has conquered those problems, lessons learned along the way, and how we're moving forward to continue making development for Spark easier within the community.

Specifically we'll discuss:* Why understanding how Spark is built helps the developer community* The tools that've been created to ease the build burden* A walk through of the `dev/run-tests(-jenkins)` scripts and how to diagnose errors

This talk will be aimed primarily at developers seeking to contribute to Spark, anyone wanting to better understand Spark's build and CI environment, or anyone (developer or otherwise) looking at the possibility of building custom Spark forks for their organization.

Brennon York will be our speaker, he is an aspiring aerobatic pilot moonlighting as a data engineer at Capital One. His true loves are distributed computing, scalable architectures, and programming languages. He's been working hard to make Spark a stronger community and inspire collaboration through his efforts on GraphX and the core build environment.