Scott Sanderson — Frank McSherry's COST and Timely Dataflow

This is a past event

28 people went

Details

"Scalability! But at What COST?" and "Naiad: A Timely Dataflow System"

In this talk, we'll look at two papers, "Scalability! But at What COST?" and "Naiad: A Timely Dataflow System". Many of the observations and arguments from "Scalability! But at What COST?" were motivated by the authors' experience of working on Naiad. Naiad's timely dataflow model has since been ported to Rust, where it sees active development.

Scalability! But at What COST?

Distributed data processing systems often aim to provide "scalability", offering users the ability to accelerate computations by taking advantage of many machines. But building a scalable system introduces significant overheads, and many systems force programmers into restricted programming models that make it hard to use efficient algorithms. In "Scalability! But at What COST?", McSherry et al. argue that researchers should measure the COST (Configuration that Outperforms a Single Thread) of their scalable systems.

Naiad: A Timely Dataflow System

Naiad "a distributed system for executing data parallel, cyclic dataflow programs" created by researchers (including the authors of the previous paper) at Microsoft Research. Naiad introduces "timely dataflow", a dataflow programming model that introduces "logical timestamps" to support iterative and incremental algorithms.

Links:

Scalability! But at What COST?

- https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf
- https://blog.acolyer.org/2015/06/05/scalability-but-at-what-cost/
- https://www.youtube.com/watch?v=6bWBEJBMNG0

Timely Dataflow

- http://delivery.acm.org/10.1145/2530000/2522738/p439-murray.pdf
- https://github.com/TimelyDataflow/timely-dataflow

Bio:

Scott Sanderson is a Principal Engineer at Quantopian, where he builds tools to help strangers from the internet make sense of complex financial data.

Please note our new location! This month's meetup will be hosted at the Microsoft New England Research and Development Center, in Kendall Square. More details to come!