Real World Data Pipeline Technologies

Moving data around may be a very challenging task, especially in high scale scenarios. Whether it is for real time or batch processing, choosing the wrong approach may turn your life into a nightmare.

Join us to hear real production approaches and technologies applied in Outbrain and Totango.

Schedule

18:00 - 18:30 Rally-up

18:30 - 19:15 "Aletheia - Outbrain’s data pipeline backbone" - Stas Levin / Outbrain

19:15 - 19:30 Short break

19:30 - 20:15 "Evolution of Data Pipeline Architecture" - Yaki Avimor / Totango

20:15 - 20:25 Short break

20:25 - 21:00 Open discussion

21:00 - ... Wrap up and drinks at the nearest bar

Abstracts

Aletheia - Outbrain’s data pipeline backbone - Stas Levin / Outbrain

Aletheia is a new framework for implementing large scale data pipelines involving multiple producers and consumers. It supports an extensible model for target endpoints, with log files and Kafka endpoints already implemented out-of-the-box, and a straightforward approach for creating new types of target endpoints. Aletheia also provides fine-grained monitoring features, supports multiple serialization formats, and lays the foundation for managing and evolving data schemas. Aletheia drives Outbrain’s batch and realtime data pipelines, handling tens of billions of messages per day produced by hundreds of clients.

Aletheia has been recently open sourced by Outbrain and is actively maintained at https://github.com/outbrain/Aletheia

Evolution of Data Pipeline architecture - Yaki Avimor / Totango

From simplistic spring/Mysql to Hadoop/spark/Luigi/elasticsearch based data pipeline evolution. This is how we are doing that. Do / Dont do , decisions, while not compromising feature development, or simply how to overhaul your car engine while you still drive at full speed.

Open Discussion?

We invite you all to take a part in this open discussion about the data pipeline in your organization. Share with the community your experience and dilemmas you had about moving data around. The conversation can span many other topics - everything goes. Join us and make yourself heard!