Design Patterns of Streaming Platforms

This is a past event

166 people went

114 5th Ave

114 5th Ave · New York, NY

How to find us

114 5th Ave, 6th Fl

Location image of event venue


*Note: The event is on 6th Fl.

6:00 - Doors & Food
6:30 - Talk 1
7:15 - Talk 2
7:45 - Wrap & Chat

Talk 1: Real-time Processing At Scale with Flink

Speaker: Dean Shaw & Max McKittrick, Data Engineer @ Capital One

Potomac is a project at Capital One that collects clickstreams. In this talk, we will deep dive into the inner workings of Potomac's Flink Clusters. We will investigate topics such as: scaling, real-time deduplication, latency vs throughput, checkpointing and failure recovery, real-time vs micro-batch output, and more!

Talk 2:

Speaker: Aravind Ramesh, Associate Staff Engineer @ DoubleVerify

DoubleVerify is a marketing measurement software company that works with some of the worlds largest brands to authenticate the quality of digital media, giving advertisers clarity and confidence in their digital investment. In this talk, we will deep dive into DV's internal streaming platform built on top of spark-streaming and discuss some useful design patterns. We will discuss topics such as autoscaling, failure recovery, application lifecycle management, and strategies to make your pipelines more testable and operable.