Building Streaming Applications with Kafka

This time Daniel Zmuda will talk about how to build a modern streaming application with Kafka. He will first give an introductory talk on Kafka and then he will show a live demo of a streaming application with Spark Structured Streaming processing data from Kafka.

Abstract:

Apache Kafka – root source of truth for real-time data pipelines

Recent years have shown that Apache Kafka is one of the most mature players in the world of real-time distributed streaming platforms. Its simple, yet novel architecture make it perfectly fitted as a root source of information whether you design event-based distributed data processing solution or just want to bring your ETLs procedures to a new level of performance. Yet, “with great power comes great responsibility”, therefore some aspects (e.g. messages idempotency) cannot be taken as granted.

During the presentation, I will present high-level architectural concepts backing Apache Kafka and then move to a practical hands-on session including a quick set-up of a Kafka cluster (introducing details about message partitioning and fault-tolerance) as well as the execution of operations through the consumer and publisher API (including consumer groups). Another part of the demo will tackle the vast capabilities of Apache Kafka integrations e.g. with Spark Streaming.

Daniel Żmuda

I am a professional that likes to evaluate upcoming ”bleeding edge” (sic!) solutions that are on the assess and trial layers of the technology radar. I have over 10 years of experience in the world of IT (from backend programming to architecture designing), in majority JEE-related one, and in parallel 8 years of constant cooperation with AGH-UST in Krakow as a lecturer of Distributed Systems Software Engineering and Internet Application Technologies. Few years ago, I also managed to gain a Ph.D. in Computer Science. After hours, devoted dad and husband that likes to rest in the bosom of nature.

Schedule:

18:45 - Get together with a few drinks

19:00 - "Introduction to Kafka" (Daniel Zmuda)

19:45 - "Live demo of setting up and running the data pipeline" (Daniel Zmuda)

20:30 - "Stuff we recently found and we think is really cool" (The Organizers)

20:40 - Food, more drinks & chatting.

Looking forward to see you at the Novomatic Forum!

----------------------------------------

This event is sponsored by NOVOMATIC

https://secure.meetupstatic.com/photos/event/7/c/e/e/600_450151982.jpeg

NOVOMATIC is the leading provider of gaming technology and casino equipment in Europe. As such we are constantly applying cutting edge technologies to develop the most innovative solutions in the industry. Sponsoring the Modern Data Science Tools Meetup is part of our effort to contribute to the creation of a strong Data Science and Machine Learning community in and around Vienna.