Kafka RDBMS bi-directional integration

Agenda:
18:00 - 18:30: Networking, Mingling, Pizza & Beer.

18:30 - 19:30: Streaming millions events per second from Kafka to RDBMS. Shani Einav, Alooma. YouTube Live Stream Event: https://goo.gl/cfyQ2R

19:30 - 20:30: Building an end-to-end streaming analytics application from RDBMS to dashboard with Kafka, Björn Rost, Pythian. YouTube Live Stream Event: https://goo.gl/ZEXApa

First talk description:
We'll talk about how to use Apache Kafka to build a data pipeline that processes and loads data from Kafka to any RDBMS in real time.
We'll hit 3 main points on the challenges of building a data pipeline:

• Transforming data in real time while making sure you still deliver exactly once
• Processing efficiently by decoupling processing from loading
• Loading data into an RDBMS while maintaining the exactly once promise
  We'll also dive into the last T in ETLT: Transforming data after you've already loaded, to hit that last layer of exactly once. All of this with a real-life example of Kafka to Redshift!

First speaker bio:
Shani is a backend Java developer who for the last past 2 years has been building a complex stream processing engine on top of Kafka. In the past, she was a part of a core backend team at Taboola. Shani holds a degree in Computer Science from the Hebrew University and is currently working towards MSc in Machine Learning at Tel Aviv University.

Second talk description:
Apache Kafka is a massively scalable message queue that is being used at more and more places connecting more and more data sources. This presentation will introduce Kafka from the perspective of a mere mortal DBA and share the experience of (and challenges with) getting events from the database to Kafka using Kafka connect including poor-man’s CDC using flashback query and traditional logical replication tools. To demonstrate how and why this is a good idea, we will build an end-to-end data processing pipeline. We will discuss how to turn changes in database state into events and stream them into Apache Kafka. We will explore the basic concepts of streaming transformations using windows and KSQL before ingesting the transformed stream in a dashboard application.

Second speaker bio:
Björn Rost is an Oracle Developer Champion, ACE Director, and one of Pythian’s top Oracle experts. A popular presenter, Björn travels the world attending technology conferences, sharing insights, and learning with his wide network of peers. Björn is attending ILOUG Tech Days 2018 in Israel.