Using events, streams and Kafka to support more autonomy and innovation

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."

— M. Conway

Organizations build systems that are rough copies of themselves, and hierarchical and siloed orgs will end up with systems that are hard to change and innovate on top of.

Creating the foundation for organizations that support autonomy and innovation are mainly rooted in organizational principles and values, however, there are architectural approaches and tools that can support the kind of qualities we seek.

In this meetup, we will be technical and look at how Event-Driven Microservices with stateful streams with Kafka as the foundation can be an enabler for autonomy, alignment, collaboration and fewer dependencies between teams.

Agenda

17:30-18:00

Registration + grab Pizza and beers. The Schibsted building is high security and you will receive a visitor badge after showing your ID to security. Please try to be here at 17:30 so we get everyone through security in time.

18:30

Access to the building will be closed for latecomers, but please try to stick to the registration time.

18:00 - 18:10

Intro and Welcome.

18:10 - 18:45

Building Event-Driven Microservices with Stateful Streams (talk + qa)

With Kai Wähner Technology Evangelist at Confluent

Event Driven Services come in many shapes and sizes from tiny event-driven functions that dip into an event stream, right through to heavy, stateful services which can facilitate request response. This practical talk makes the case for building this style of system using Stream Processing tools, defining a microservices architecture and leveraging Apache Kafka. We also walk through a number of patterns for how we actually put these things together to enable independent teams and autonomous development of microservices.

18:45-18:55 Break

18:55 - 19:30

Apache Kafka in the global Schibsted Data Platform (talk + qa)

With Fredrik Vraalsen, Software Engineer at Schibsted Products & Technology

Schibsted is setting up a new streaming platform based on Kafka and Kafka Streams. With this new platform, we aim to deliver better performance and enable new features, such as self-serve for our data consumers. In this talk, we will present some of the ways this new platform enables collaboration across Schibsted, as well as some of the challenges we are facing.

A number of collaborations with various teams in Schibsted are underway to build projects on top of this new platform. Examples of such collaborations are projects geared towards building tools for visualization of user behaviour and experimentation.

Our long-term goal is to provide a self-serve platform for real-time processing of data, enabling our data users to quickly create new data-driven applications. Data and analytics is a central part of Schibsted’s strategy, and we believe the streaming platform will play a significant role in building a global data-driven organisation.

19:30 - 20:30 Mingle/Networking