Real Time Analytics with Complex Event Processing

• Agenda

6:00 - 6:50 Food, drinks & chat

6:50pm: Seed Zeng and John Meichle
Real Time Analytics with Complex Event Processing

How do you build a real-time analytics system that deduplicates over a million events a second with cardinality in the billion range, and also updates over a million counters a second? How do you scale such a system to keep up with the exponential growth of workload year over year? In this talk, we will cover the requirements for real-time analytics, the technical challenges that come with it, and the migration of Klaviyo's real-time analytics system from Python to Flink. We reduced our AWS EC2 instances from over 100 nodes to 15. The system handled more than 75,000 events per second during 2018 Black Friday.

Speakers:

Seed Zeng, Software Engineer
Seed Zeng joined Klaviyo in 2017. He has been working on the event processing pipeline and data storage infrastructure. Seed has lived through two Black Friday Cyber Monday at Klaviyo, contributing to the scaling effort in many key infrastructure systems such UUID assigment, message queues, Redis clusters, and the real-time analytics system. Seed graduated with a Bachelor's degree in Computer Science from Washington University in St. Louis.

John Meichle, Senior Site Reliability Engineer
John Meichle is a Senior Site Reliability Engineer at Klaviyo. He is passionate about infrastructure, automation, scrum, and observability. His focus at Klaviyo has been to sustainably grow the engineering organization by building new infrastructure patterns, scaling the Klaviyo's metrics stack to over a million metrics, and innovating Cassandra operations. Prior to joining Klaviyo, John led an Automation team at Acquia and designed a workflow framework for operational processes across 10,000+ AWS instances. When not focusing on scaling systems, John can be found turning wrenches on cars or playing games with friends.