addressalign-toparrow-leftarrow-leftarrow-right-10x10arrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscontroller-playcredit-cardcrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobe--smallglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1languagelaunch-new-window--smalllight-bulblinklocation-pinlockm-swarmSearchmailmediummessagesminusmobilemoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonprintShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahooyoutube

Matei Zaharia Keynotes Scale By the Bay 2017 on Spark and Weld

From: Alexy K.
Sent on: Tuesday, November 7, 2017 10:52 AM

Dear SF Scala members -- as you know, Matei Zaharia, the creator of Apache Spark, has chosen Scala, a functional programming language, to implement it, with one of the benefits being composability.  Composable abstractions allow for creation of comprehensive data pipelines and better reasoning about their behavior and performance.  We are fortunate to add Matei's keynote to Scale By the Bay this year on 11/17 at Twitter HQ.  Matei spoke at Scala By the Bay many times and is a great friend of the community.  Below is Matei's keynote abstract.

Composable Parallel Processing in Apache Spark and Weld

The main reason people are productive writing software is composability: engineers can take libraries and functions written by other developers and easily combine them into a program. However, composability has taken a back seat in early parallel processing APIs. For example, composing MapReduce jobs required writing the output of every job to a file, which was both error-prone and slow. Apache Spark helped simplify cluster programming largely because it enabled efficient composition of parallel functions, leading to a large standard library and high-level APIs in various languages. In this talk, I'll explain how composability has evolved in Spark's newer APIs, and I’ll present Weld, a new research project I'm leading at Stanford to enable much richer composition of software on emerging parallel hardware (multicores, GPUs, etc). Systems like Weld and Spark will allow engineers to focus on building their application rather than the intricacies of parallel hardware, and might represent one of the best ways we have to tame the ever-diversifying hardware landscape.

We also added more panelists and more talks, as well as fireside chats, at Scale By the Bay.

Fireside chat with Ben Hindman, the creator of Apache Mesos, and Ian Downes, the current Mesos lead at Twitter, is on 11/16, during the Happy Hour, at Twitter SF.  It is worth remembering that Spark originally was a test application for Mesos!

Fireside chat with Evan Weaver, the original data lead at Twitter and now Fauna founder, andf Boaz Avital, the current data lead at Twitter, is on 11/17, during the Happy Hour, also at Twitter SF.

The Istio/Envoy/gRPC workshop is hosted and sponsored and co-run by Google at Google Launchpad.  It is getting quite full.

We blog with speakers at  It includes the interviews with the Legends of Twitter, folks who started companies based on the Twitter architecture stack, and who will reunite on the Legends of Twitter and Beyond: Real-World Architectures panel on 11/16.

The Functional Programming for Machine Learning panel added Takt and Criteo members.  Oscar Boykin, the creator of Scalding and now at Stripe, is on the panel, moderated by Vitaly Gordon, who leads Salesforce Einstein Data Engineering and Data Science.

The conference itself has less than 50 Late Bird passes left overall.  Reserve your seat today with the code SFSCALA15.


People in this
group are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy