Distributed, high-throughput, persistent real-time messaging with Apache Kafka

Pizza and soda for the meetup is being provided by LinkedIn, the company originally responsible for building Kafka. Thanks @jaykreps for the generosity.

Apache Kafka is (from the project page):

... a distributed publish-subscribe messaging system. It is designed to support the following

  • Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
  • High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
  • Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
  • Support for parallel data load into Hadoop.


Talk Outline:

  • The design of Apache Kafka
  • Some hand-wavy diagrams of the linked-in and foursquare topologies
  • Why I really like Kafka
  • Kafka vs other pub/sub and queue systems (like redis, rabbit)
  • Kafka vs other log persistence systems (like flume / scribe)
  • Takeaways from the production deployment at foursquare



Join or login to comment.

  • Kyle N.

    Was really looking forward to meeting everyone, but I have a sick fiance at home I need to take care of. Hopefully I'll see you all at the next one.

    February 26, 2013

  • Andrew B.

    Can't make it. Have fun!

    February 26, 2013

  • greg k.

    looking forward to it!

    February 24, 2013

  • Jim H.

    bummed that I can't make it. will be there in the future!

    February 20, 2013

32 went

Our Sponsors

  • Cloudera

    Cloudera is the general sponsor of Big Data Madison.

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more

I started the group because there wasn't any other type of group like this. I've met some great folks in the group who have become close friends and have also met some amazing business owners.

Bill, started New York City Gay Craft Beer Lovers

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy