PWL #59: Lightweight Asynchronous Snapshots for Distributed Dataflows

Papers We Love @ Seattle
Papers We Love @ Seattle
Public group

Every 1st Thursday of the month

Location image of event venue

Details

Details
• What we'll do
Lightning Talk: A very special guest David Murray will be presenting on the Convoy Phenomenon (https://jimgray.azurewebsites.net/papers/Convoy%20Phenomenon%20RJ%202516.pdf)

A congestion phenomenon on high-traffic locks is described
and a non-FIFO strategy to eliminate such congestion is presented.

Main Event: Max Payton will be presentingLightweight Asynchronous Snapshots for Distributed Dataflows (https://arxiv.org/pdf/1506.08603.pdf)

Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots.

• Important to know

Big ups to Comcast for hosting this month!

As a chapter of Papers We Love we abide by and enforce the PWL Code of Conduct (https://github.com/papers-we-love/seattle/blob/master/code-of-conduct.md) at our events. Please give it a read, plan on acting like an adult, and involve one of the organizers if you need help.

Stop slacking and join us in the #seattle channel at https://papersweloveslack.herokuapp.com!

If you have a paper you'd like to present, or even just a mini, please hit up one of the organizers :) We're always looking for more presenters.