addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupsimageimagesinstagramlinklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1outlookpersonJoin Group on CardStartprice-ribbonShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruseryahoo

Running Kafka in production

  • Needs a location

  • Near Brussels, Mechelen or Antwerp
  • I have postponed the Meetup to January 2017 (e.g. 18 Jan 2017).

    I prefer to have 2 or 3 speakers and was only able to find one speaker that can actually prepare and present on short notice. If you have an interesting story on Kafka, please propose an abstract, so we can have a few speakers on the Meetup.

    First Speaker confirmed:

    Hatem Mostafa of CoScale on  "from[masked] requests per day to[masked] requests per second with Kafka"


    In CoScale we worked on a project to evaluate how we can scale our system to from[masked] requests per day to[masked] requests per second. This talk will explain how we did that using a stream processing approach, as well as elaborate in depth the design and which technologies and tools we used and why. Also the bottlenecks that we faced and how we solved them.

    Still looking for 1 or 2 additional speakers for the evening.

    I would like to organise a next Meetup on an evening begin 2017 about "Running Kafka in production".

    If you have a set-up running Kafka in production or know people that do, please take contact so we can arrange a presentation at this Meetup.

    Some questions that would really interest me to get answered are:

    * Why did you start with Apache Kafka ?

    * What are the alternatives you considered and what where the trade-offs (Rabbit MQ, zeroMQ, ActiveMQ, Kinesis, other PaaS messaging, cloud hosted Kafka, ...) ?

    * How did you organise the set-up and the maintenance of the cluster(s) ?

    * What is the team size and organisation (separate squad, part of INFRA, distributed over the squads implementing business stories)

    * What is the cluster size, partitions, retention times, retention policies ? What where the trade-offs for more or less partitions, and other technical choices ?

    * How much data at rest, how many message per second, how many topics, how many producers and consumers ?

    * How do you manage version updates ?

    * Is the Kafka cluster running on-premise or in the cloud ?

    * How to manage multi-datacenter (on-premise or in the cloud, DC replication) ?

    * How to manage "back-up" and/or "disaster recovery" ?

    As I am personally not aware of many Kafka production set-ups yet in Belgium (I know a few that I will contact in private), I am looking very much forward to any contributions you may have.


    Peter Vandenabeele



Join or login to comment.

  • Raph H.

    Hi Peter, Ixor would be happy to host. drinks included.

    1 · November 22

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy