Using Apache Kafka as a Large Scale Distributed Message Bus

AppsFlyer is successfully using Apache Kafka as its main message bus to handle billions of events each day in a reliable and stable fashion. Kafka has allowed AppsFlyer to easily scale our system while our event load had grown more than tenfold.

In this edition of "Coding with AppsFlyer", our own Adi Belan will walk you through best practices of producing/consuming to/from Kafka, highlighting aspects of Kafka's durability and parallelism options, each participant will try his/her hands in implementing a complete flow using Kafka consumers and publishers.

In this session:

A classic example of how AppsFlyer uses Kafka in production is our in-app events sorting service, which consumes messages from endless (really, it is endless :)) Kafka topic. Each message is processed according to a result of a database query and then published to one of a few possible Kafka topics.

In this session, each Coder will (try and) complete an exercise very similar to the in-app events flow described above. We'll use a Kafka consumer (in a programming language of your choice) to receive messages from an endless stream, create some basic logic and publish the messages according to the logic to several other Kafka topics. We will provide examples in Scala and Clojure.

We'll also review some of the built-in tools Kafka provides including, such as the console consumer, topic describer and consumer lag checker. In addition, we will demonstrate some visualization tools we use in AppsFlyer to help us monitor and test our services.