Running Kafka in production and Streamsets

18:30 - 19:00 Arrival and small snack offered by De Persgroep

19:00 - 19:10 Short intro on De Persgroep

19:10 - 19:45 "Kafka Streams and Kafka as a simplification for stream processing"

Abstract:

This talk will introduce Kafka Streams and explain why Apache Kafka is a great option and simplification for stream processing.

Bio:

Paolo Castagna is a Senior Sales Engineer at Confluent. His background is on 'big data' and he has, first hand, seen the shift happening in the industry from batch to stream processing and from big data to fast data.

19:45 - 20:20 "from 500 000 requests per day to 1000 000 (1 million) requests per second with Kafka" by Hatem Mostafa of CoScale

Abstract:

In CoScale we worked on a project to evaluate how we can scale our system to from 500 000 requests per day to 1 000 000 requests per second. This talk will explain how we did that using a stream processing approach, as well as elaborate in depth the design and which technologies and tools we used and why. Also the bottlenecks that we faced and how we solved them.

20:20 - 20:30 Pauze

20:30 - 21:00 "Rapid data ingestion pipelines with StreamSets" by Rob Gibbon of Big Industries

Abstract:

In this talk Rob Gibbon will turn the microscope on StreamSets, a new, open source streaming data ingestion system for the Hadoop ecosystem and friends. Rob will give us an overview of this useful tool, guide us through the process of developing a data ingestion pipeline, and look at options for extending the base functionality.

Bio:

Robert Gibbon - Managing Partner Big Industries Robert is a technical architect with hands on knowledge of Big Data system design, build and operation. He has gained his experience building solutions in varied domains with organisations ranging from upstart to blue chip and he can quickly adapt to changing needs.

21:00 - 21:45 Drinks