Streaming Data At Scale

Streaming Data At Scale

Streaming applications enable real-time analytics and decisions. In the Big Data era, the challenges arise not only for low-latency, high-throughput but also for high-volumes of data. Come and learn about how companies build their large scale streaming applications, the challenges they faced and the lessons learned in the process!

Schedule:

6:30 - Doors & Food

7:00 - Talk 1

7:45 - Talk 2

8:30 - Wrap & Chat

Talk 1: Building Low Latency Feedback Loops at High Scale

Speaker: Jacie Fan, Senior Software Engineer & Aaron Dornbrand Lo, Data Services Team Lead, Appnexus

Abstract: This talk will cover how we use commercial and open source tools for our streaming use case. Jacie will discuss the importance of processing data with low latency and give an overview of our Lambda architecture. Aaron will do a deep dive into our rapid feedback loop which involves hundreds of bidders across 5 data centers in 3 continents, and he will talk through challenges we’ve encountered while scaling.

Talk 2: Data Quality Monitoring in Realtime… at Scale

Speaker: Alexis Seigneurin, Big Data Engineer, Ippon USA

Abstract: Kafka has become extremely popular to stream data, but it imposes very little constraints over the format of the data that is being streamed. As we wanted all of our Data Engineers and Data Scientists to use the data in our Kafka clusters, we soon faced the challenge of keeping the quality of our data to its highest. We developed a tool to monitor the quality of the streams in realtime, and we had to make it scalable and fault tolerant. In this talk, we will see the technical difficulties we encountered, and how we went through a major rewrite of the application to make it scale.