Robust Stream Processing with Apache Flink with Jamie Grier of data Artisans

Robust Stream Processing with Apache Flink

In this hands-on talk and demonstration I'll give a very short introduction to stream processing and then dive into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible. We'll focus on correctness and robustness in stream processing.

During this live demo we'll be developing a realtime analytics application and modifying it on the fly based on the topics we're working though. We'll exercise Flink's unique features, demonstrate fault-recovery, clearly explain and demonstrate why Event Time is such an important concept in robust stateful stream processing and talk about and demonstrate the features you need in a stream processor to do robust stateful stream processing in production.

We'll also use a realtime analytics dashboard to visualize the results we're computing in realtime. This will allow us to easily see the effects of the code we're developing as we go along.

Some of the topics covered will be:

• Apache Flink
• Stateful Stream Processing
• Event Time vs. Processing Time
• Fault tolerance
• State management in the face of faults
• Savepoints
• Data re-processing

Jamie Grier is Director of Applications Engineering at data Artisans where he’s extremely excited to be able to help others realize the potential of Apache Flink in their own projects. His goal is to help others design systems to solve challenging problems in the real world.

Jamie has been working in the field of streaming computation for the last decade. This has spanned everything from ultra-high-performance video stream acquisition and processing to social media analytics.

Prior to joining data Artisans, Jamie was at Twitter working on rethinking the realtime analytics stack with the goals of making it much more efficient and also capable of computing accurate results in real-time without relying on the “Lambda Architecture” for correctness.

Before Twitter Jamie was one of the lead engineers at Gnip building their social media streaming, filtering and delivery system and before that was the lead engineer at Boulder Imaging working on systems that could ingest and process greater than 1 GB/sec of streaming video on a single machine.

Jamie is interested in streaming computation and mechanically sympathetic software architectures. He is particularly interested in building systems that are both high performance and highly scalable and his favorite quote is “You can have a second computer once you’ve shown you know how to use the first one”.