Intro to Spark 0.7: Python API and Spark Streaming

The Spark team has been hard at work on two big features for release 0.7: PySpark, which adds a Python API to Spark, and an alpha release of Spark Streaming, which adds easy-to-use stream processing functionality. With Spark 0.7 coming out very soon, this meetup will introduce attendees to the new features. We're going to have two presenters:

1) Josh Rosen will show how to use the Python API. PySpark provides almost all of the features of Spark to Python programmers, both in standalone programs and from the python and IPython interactive shells. It works with the standard CPython engine, letting you use native libraries like NumPy and SciPy in your Spark programs. It also handles shipping functions to the cluster just like in Java and Scala. We encourage you to invite your Python friends to learn about it!

2) Tathagata Das (TD) will cover Spark Streaming, a new extension of Spark to do near-real-time stream processing that will be available as an alpha in Spark 0.7. We introduced Spark Streaming from a research perspective last summer, but this talk will show what the complete API looks like, and discuss issues such as data input sources and fault tolerance. TD will also cover several applications, including a prototype implemented at Conviva to take Conviva's Hadoop-based batch analytics pipeline (a series of MapReduce jobs that normally sees 5-10 minutes of latency) and run the same Hadoop code on Spark Streaming with 2-second latency. This ability to run the same code in both batch and streaming settings is one of the reasons why we're very excited about Spark Streaming.

Conviva graciously offered to host this meetup at its San Mateo office. Food will be provided. Doors open at 6:30, with talks starting at 7.

Join or login to comment.

  • Sam B.

    Is there any video of the event anyhow?

    February 27, 2013

  • Kalpit S.

    Had been waiting for this talk since a few months....It was great...The content was clear and detailed....Good job TD, Matei and others !

    February 26, 2013

    • Tathagata D.

      Thanks. You should try PySpark and Spark Streaming!

      February 27, 2013

  • Haidar H.

    any plans to do spark hackathon?

    February 21, 2013

    • Sam B.

      We'll be doing a Spark/shark/Spark streaming. And we have some pythonistas we'll try the python API as well.

      February 23, 2013

    • Tathagata D.

      Great! I think a good starting material will be the AMP Camp material (see http://ampcamp.berkel...­). That covers Spark and Shark. We are in the process of extending this material to cover Spark Streaming as well for the Strata hands-on tutorial this week. Hopefully we will be able to give that out to people.

      February 25, 2013

  • Matei Z.

    Thanks everyone for coming! I've posted the slides at

    February 22, 2013

  • Paul P.

    Great presentation last night. Just wondering if you guys can share your slides ?

    February 22, 2013

  • Stoney V.

    Great presentations Josh and Tdas :) Every release brings more value.

    February 22, 2013

  • Puneet

    Very insightful!

    February 21, 2013

  • Emre

    I was new to Spark so I appreciated the background, and simple examples. Live demonstrations were the best part.

    February 21, 2013

  • Larry M.

    Can't get down there.

    February 21, 2013

  • Emre

    Psyched about this event!

    1 · February 18, 2013

  • Denny L.

    Sorry for missing this one, eh?!

    February 17, 2013

  • Ajith J.

    interested in data science

    February 17, 2013

  • Sam B.

    Hopefully, I'll ne on SF for the next meetup. Have fun.

    February 16, 2013

  • Stoney V.

    Super excited about PySpark and Spark streaming. I intend to record video and audio of the meetup. It would be great if presenters would be ready to do screen capture of any code browsing so that it can be included with any presentation slides. ( Snow Leopard, QuickTime Player, New Screen Recording ) Ctl key zooms in.

    2 · February 15, 2013

  • A former member
    A former member

    Hi, It would be great if you could capture this on video

    February 15, 2013

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy