An evening of Apache Kafka

It's been a long time since the last meetup but hopefully worth the wait.

An evening of all things Apache Kafka. We have two great talks. Confluent, introducing their new, Kafka SQL and Rental Cars with some real world tales from the trenches.

Agenda

6.30pm Networking (Beer and Pizza)

7.00pm Welcome

7.05pm Introducing KSQL: Open Source Streaming SQL for Apache Kafka
By Tom Scott, Customer Operations Engineer at Confluent

KSQL is an open source, streaming SQL engine that enables stream processing with Apache Kafka. KSQL makes it easy to read, write, and process streaming data in real-time, at scale, using SQL-like semantics. It offers an easy way to express stream processing transformations as an alternative to writing an application in a programming language such as Java or Python.

In this talk we will create a sample KSQL application and use it to achieve some real-world stream processing tasks that would otherwise require extensive development. We will also explore the Kafka Streams technology that underpins KSQL.

8.00pm Kafka and Spark integrations – real life lessons learnt
By Matt Slack, Data Architect at Rental cars

Kafka/Spark Streaming from the trenches! Some lessons learnt from real world scenarios, ingesting high volume, often poor quality streaming data into a Hadoop data lake. Covering topics such as:

• Handling diverging, complex event schemas

• Optimising Spark streaming to perform well with unpredictable event size and volume

• Managing and monitoring data growth in Hadoop

• Working with different (sometimes conflicting) Kafka and Spark versions

8.25pm Closing remarks

8.30 - 9.00pm

Networking

Presenter's Biography:

Tom Scott

Customer Operations Engineer at Confluent interested in all things big data and distributed.

Matt Slack

Data Architect at Rental Cars. Most of his time is spent developing and evangelising the Hadoop estate, and trying to keep on top of the latest and greatest streaming developments, both on-premise and cloud.

Prior to RentalCars, he worked with Hadoop in several different guises for 3 years, mostly in Financial Services.