Spark 1.1 and Advanced Spark Tips

In this meetup, Patrick Wendell from Databricks will be speaking about Spark's upcoming 1.1 release. This release includes significant extensions to Spark's SQL, MLlib and Streaming libraries. It also adds several performance and robustness improvements in Spark's core engine.

In addition, since we're talking about Spark internals, we'll also cover some more advanced concepts regarding Spark's internal execution to explain what has changed. This talk will focus on providing lower level details to help users who are performance-testing or debugging Spark, or trying out new Spark applications.


Patrick Wendell is a committer and PMC member on Spark, as well as the release manager for the Spark 1.0 and 1.1 releases.


Space is limited, so we'll only be able to provide admission to people confirmed on the RSVP "YES" list, and for building security we'll need you to provide your full name and an email address with your RSVP. Please register via, with your full name, before 11 PM on August 26th.


We'll be in Apple building DA3, in the theatre.

The room will open at 6, with talks starting at 7. To avoid parking problems, you may wish to arrive at 6:15 or later.

Join or login to comment.

  • Wenjing C.

    Very interested in this thread of discussion. I too thought that question wasn't convincingly addressed during the meetup. Paul, would you mind sharing more about the 'emerging reactive stream API'?

    September 5

  • Jaka J.

    Great meetup!

    I was not entirely convinced by the answer to the (great!) question about why Spark core and streaming APIs need to be different.

    If someone wants to respond, I'm sure there are more of us curious people out there :)

    1 · August 28

    • Paul S.

      Can you maybe be more specific? To be honest, I wasn't convinced by the question. ;-) That is, the structure of DStreams makes sense to me: it's a discretized stream of RDDs. Now, I agree it's not the best stream API and it will be interesting to see how it holds up in comparison to the emerging reactive stream API, but that's a distinct question from why Spark core and Spark streaming are somewhat different.

      1 · September 5

  • Burt P.

    One of the best Spark meetups I've attended! Amazing job by Patrick, especially clearly describing the issues surrounding Spark's "shuffle". It is really awesome to hear about Spark internals from Patrick. It helps a great deal.

    Josh and TD did a fantastic job on fielding questions. The object serialization that occurs across the network was fascinating to learn about.

    Thank you Databricks for your magnificent support of the Spark community.

    Lastly, thanks to Apple for hosting us.

    2 · August 28

  • Patrick W.

    I added slides in PDF and PPTX - thanks to everyone who came and to Apple for hosting!

    6 · August 28

    • Jorge M.

      Great presentation Patrick!

      August 28

  • Michael F.

    Best venue, visuals, audio, food, speakers and questions ever for me.
    Equating Hadoop to 'Yucky'
    clearly exposed the final nail in the coffin.
    May Hadoop RIP :)

    1 · August 28

  • Alexis R.

    Great preso / updates on Spark!

    August 27

  • Nilesh B.

    Good updates...

    August 27

  • Nilesh B.

    Good updates...

    August 27

  • Liz B.

    When you arrive, please come to the lobby at the back of our building.

    August 27

  • Minh

    Some people canceled and now there are 20 open spots. This RSV is closed. I would like to go can someone take down the name?

    1 · August 27

  • Stanley W.

    how can I find out what spot I hold on the waiting list?

    August 27

  • Aniket A.

    I am on the waiting list in first 15; and I see there are 15 spots left.
    Does that mean I will be able to attend it?

    August 27

  • monir M.

    When I RSVP'd for the event, it asked me how many guests I will bring and I put 2. One of my guests will not accompany me so only one guest will be there. Organizers, see if this opens up a slot for one more person.

    August 27

    • monir M.

      Ah, I tried to change the RSVP to note my change to only one guest. and the System accidentally put me in wait list. As RSVPs are closed, I assume my original RSVP still holds and I will come to the event with the guest.

      August 27

  • Premchand

    Hey I just saw this meetup group and would really love to attend the session today and I still see you have 10 spots left. If that is the case, could you please let me in to today's session?

    August 27

  • Steven S.

    Would really like to be there in Cupertino! I only have the opportunity at the moment to attend virtually (via Skype or Hangouts or whatever). I noticed that you maxed out your physical attendees. Would someone be willing to set up a phone/tablet/laptop to which folks like myself can Skype into? Not sure what the limit is of a conference call but might be worth a short since there's so much interest!

    August 26

  • Mohammed G.

    Are there plans to record this session and upload the video/slides?

    August 25

    • Matei Z.

      No recording for this one unfortunately, but the slides will be posted after.

      August 26

  • Kun H.

    Hello everyone, this is the first time I apply for a meetup, I wanna know if not being on Going List means that I can not attend the meetup? Thank u guys:)

    August 25

    • Matei Z.

      In this case, unfortunately, if you don't make it into the "going" list you won't be able to get in, because of limited capacity. Future meetups should have more space.

      August 25

    • Kun H.

      OK, thank u so much!

      August 25

  • Subodh N.

    Why are all these events in South bay. After all Databricks is located in Berkeley couldn't do some meetups in East bay?

    3 · August 20

    • Patrick N.

      SF would be more convenient than Berkeley for most of us. Most of Scala developers are located in Mid-peninsula and So. bay.

      August 25

    • Siamak K.

      Subodh: I have the same reaction when I see so many great meetups in San Francisco that are hard for me to get to :-)

      August 25

  • parimal

    I am on waitlist. Any chance of having meetup in bigger conf room

    3 · August 24

  • Shivani R.

    It would be nice to have a presentation uploaded before hand, so people can decided whether to attend or not based on the level of detail that suits them

    2 · August 22

  • Justin Y.

    is there streaming?

    3 · August 20

    • Matei Z.

      Unfortunately, we won't be able to provide streaming for this one. Likely to have it for future meetups though.

      2 · August 21

  • Burt P.

    Yea Patrick! Woohoo, Spark internals! Thank you!

    This is going to be awesome!

    August 20

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy