Kafka @NORD/LB: Basic principles on implementing a Kafka-based API

Dear Streamers,

Join us for an on-site Apache Kafka® Meetup on Thursday, November0 03rd from 6:00 pm in Hanover kindly hosted by our friends at NORD/LB. The address, agenda and speaker information can be found below. See you there!

***DISCLAIMER****
BY ATTENDING THIS EVENT IN PERSON, you acknowledge that risk includes possible exposure to and illness from infectious diseases including COVID-19, and accept responsibility for this, if it occurs.
***

Agenda:
6:00 pm: Doors open + Pizza
6:30 pm-7:15 pm: Basic principles on implementing a Kafka-based API - Erik Schumann and Sven Wilbert, NORD/LB
7:15pm-7:45pm: Think of ksqlDB before using KafkaStreams - Stream Designer Edition - Dominique Ronde + Patrick Neff, Confluent
7:45pm-7:55pm: Building a community in Hanover
7:55pm -9:00 pm: Pizza, Q&A + Networking

Speakers & Abstracts:

• Basic principles on implementing a Kafka-based API
• Erik Schumann & Sven Wilbert, NORD/LB
• Abstract:

In order to achieve our best-of-breed approach at NORD/LB, we use Kafka as an event-streaming platform. Additionally, the API first concept is an integral part of our architectural principles. However, application-based APIs can cause problems during implementation as well as operation: Expensive interface modules of proprietary software can exceed the generated value or legacy systems can be too slow. Accordingly, we evaluated in a proof of concept whether we can also implement in-house APIs in a Kafka-based approach by using data from our business partner system.

In the first attempt, we used Kafka ksqlDB like a relational database yet due to slow responses, this method failed quickly. Considering that Kafka is based on a key-value store and therefore all non-key-based queries triggering a full-table scan, this was however a likely outcome. Nevertheless, to a limited extend, this can be handled with materialised views for predictable queries. As in our case queries are not key-based and cannot be pre-calculated, we decided to use an external database – the advised method in cases like this. This database allowed us to use our own API for the desired communication with Kafka.

During this proof of concept for a Kafka-based API we learned about Kafka's limitations and understood that Kafka requires a new way of thinking: Data in Motion

• Think of ksqlDB before using KafkaStreams - Stream Designer Edition
• Dominique Ronde + Patrick Neff, Confluent
• Abstract:

Stream Designer - the latest Confluent Cloud features has been released at Current 2022! In this session, we present a live demo of the Stream Designer on a typical IoT data pipeline based on ksqlDB. We interactively apply data transformation, such as projection and aggregation, and highlight different aspects of the feature.

The presentation is a follow-up on “Think of ksqlDB before using KafkaStreams” which was presented as a meetup in March this year.

-------