Name: Journey of two streaming frameworks - Spark Streaming and Kafka Streams
Start: 2019-09-15T18:00:00+03:00
End: 2019-09-15T21:00:00+03:00
Location: Imperva

18:00 - 18:30 - Mingling
18:30 - 19:30 - Realtime Data Pipelines Using Spark Streaming - Yulia Stolin - Recommendation Group Architect at Outbrain
19:30 - 20:15 - How we built our time-series and tops processing system based on Kafka-Streams, from the naïve approach to the working one - Omer Litov Engineering Manager Data Infrastructure @ Imperva

Title:
Realtime Data Pipelines Using Spark Streaming - Yulia Stolin - Recommendation Group Architect at Outbrain

Abstract:
Having near real-time inputs is extremely important, when you running high scale recommendation systems.
During this session, I will present our journey from batch-based to real-time analytics.
We implemented a data-pipeline using Spark Streaming on top of Kafka for real-time, accurate and precise decision making. I will introduce the main components of our architecture, data managements, and lesson learned from the process.
Finally we will overview different use-cases and their architecture such as
* Building and running real-time predictive analytics using Contextual Multi-Armed Bandit Models for UI ABTest optimisation.
* Running predictive CTR (click through rate) estimation based on real-time data inputs using weighted linear regression models.
* Building the analytics reports combining real-time data

In the end of the session, you will be familiar with Lambda Architecture and Streaming concepts.
You will also learn how to use Spark to combine real-time and batch analytics, and become more familiar with Spark’s capabilities.

Bio:
I have 15 years of hands-on experience in software architecture, specialising in building high volume, scalable, high-performance, distributed data systems. Expertise in BigData, NoSQL, Architecture and Development. I’m working at Outbrain as a Recommendation Group Architect. I have two sons and like to spend my free time traveling with the family, playing tennis, and swimming

Title:
How we built our time-series and tops processing system based on Kafka-Streams, from the naïve approach to the working one - Omer Litov Engineering Manager Data Infrastructure @ Imperva

Abstract:
We were faced with the challenge of building a backend system for a new type of dashboards. The new dashboards consist of both time-series graphs, and tops tables (with virtually unlimited key range). We already had a working proprietary backend system for time-series processing, but it didn’t support unlimited key ranges, and started to show performance issues. Instead of modifying the existing system, we decided to try a new and promising framework – Kafka Streams.
Most big data framework are very complicated, requiring big efforts to set up correctly and maintain. That makes the idea behind Kafka-Streams very appealing, it is a simple all-in-one library, running as a dependency withing your microservice. No need to set up complicated processing clusters, schedulers, or even a DB, everything is inside. Combined with it’s super simple API, you can set up a simple Kafka-Streams application within minutes.
That is the theory anyway, the reality turns out to be a bit more complex. Due to different issues I will describe, we avoided using Kafka-Streams high level API, and switched to a lower level API, and setting up a dedicated Cassandra DB. After all the modifications we did, we ended up with a very robust and performant system.

Bio:
Omer Litov Engineering Manager Data Infrastructure @ Imperva

shlomi hassan

Big Things

Technology

Open Source

Apache Druid

Hadoop

Big Data

High Scalability Computing

MapReduce

Machine Learning

Data Analytics

NoSQL

Data Science

Elasticsearch

Apache Lucene

Data Mining

Apache Spark

Journey of two streaming frameworks - Spark Streaming and Kafka Streams

Imperva

Share

Big Things

Journey of two streaming frameworks - Spark Streaming and Kafka Streams

Big Things

Details

Members are also interested in