Getting to know Apache Kafka

We are glad to announce the first Data Engineering Oslo Meetup! Our focus is on Apache Kafka with two talks:

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform
by Kai Wähner from Confluent

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases.

Kafka and Kafka Streams in the global Schibsted Data Platform
by Fredrik Vraalsen from Schibsted

In this talk we will present how we in Schibsted have set up a new global streaming data platform using Kafka and Kafka Streams, replacing a homegrown solution based on Kinesis and micro batches in Amazon S3. Schibsted is a global media and marketplaces company with over 200 million users across the world.

With our new streaming platform, we aim to deliver better performance and enable new features, such as self-serve for our data consumers. In this talk, we will present some of the ways this new platform enables collaboration across Schibsted, as well as some of the challenges we have faced.

A number of collaborations with various teams in Schibsted are underway to build projects on top of this new platform. Examples include building tools for experimentation on our sites, visualisation of user behaviour, and data quality checks.

Our long-term goal is to provide a self-serve platform for real-time processing of data, enabling our data users to quickly create new data-driven applications. Data and analytics is a central part of Schibsted's strategy, and we believe the streaming platform will play a significant role in building a global data-driven organisation.