addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupsimageimagesinstagramlinklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1outlookpersonJoin Group on CardStartprice-ribbonImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruseryahoo

Realtime Distributed Message Processing at Scale with NSQ

Matt Reiferson and Jehiah Czebotar from will be talking about Realtime Distributed Message Processing.

About NSQ

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.
It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.
It is the backbone of bitly's infrastructure, powering realtime data analysis and feeding offline batch processes.


About the speakers.

Matt is an Engineer @bitly, one of the main architects behind NSQ.

Matt Reiferson is a software engineer, lunch instructor, and fan of tv's Bob Ross. He is most known as the creator and host of The Joy of Coding in DOS, a television program that ran for twelve years on PBS stations in the United States.


Jehiah is a co-author of NSQ. When not painstakingly shortening urls by hand, Jehiah can be found riding trains, learning Amharic, or building iPhone apps to help others do both of those things more efficiently.

A lover of data, he has been known to compile amazing personal annual reports documenting the details of his day to day life that most people take for granted, which you can see here:

It's a little known fact that Jehiah's sleep, not his code, powers bitly.

Join or login to comment.

  • A former member
    A former member

    Hi all, some of you emailed me that you didn't receive the video link. Here it is again:

    June 24, 2013

  • A former member
    A former member

    Let us know what topics you are interested in for future talks.

    May 16, 2013

    • Paul C.

      Yeah I would love to hear a talk on data journalism.

      May 18, 2013

    • Yangbo D.

      Note that Narrative Science prefer the term 'narrative scientist' to 'data scientist'. Here is some background from their CTO:­

      May 18, 2013

  • David

    Don't mean to be irreverent but the product design is nowhere near existing commercial offerings. Looked to me like an attempt to reinvent the bicycle by amateurs. Message loss and performance bottlenecks are almost inevitable in real life. The presentation itself was not bad.

    May 15, 2013

    • Matt R.

      Multicast isn't available in many environments, unfortunately (AWS being a notable example)

      May 16, 2013

    • David

      Not too familiar with AWS, I thought that you could build your own virtual network on EC2 with M/C support. I could be wrong though.

      May 16, 2013

  • Otis G.

    Looks like good piece of software. Nice demo.

    May 15, 2013

  • Richard D.

    Great presentation on a really cool way of managing messages at a large scale and in real-time.

    May 15, 2013

  • A former member
    A former member

    Will publish the video and slides in the next couple of days.

    May 15, 2013

  • Norman G.

    Looks like a super interesting talk. I'll definitely catch up on the video feed.

    May 15, 2013

  • Rick S.

    Not gonna be able to make it. Hope to see you all at the next one

    May 14, 2013

  • Drew W.

    Had a conflict - wish I could have made it!

    May 14, 2013

  • A former member
    A former member

    We have exciting news today!
    Jeniah Czebotar, co-author of NSQ, will be joining Matt in the NSQ talk this Wednesday!
    Look forward to seeing you all.

    May 13, 2013

  • Vincent D.

    Nice topic! I'm coming.

    May 10, 2013

  • Otis G.

    Does Bitly plan on open-sourcing NSQ and if so when?

    April 9, 2013

  • Geoff G


    April 8, 2013

Our Sponsors

  • Hakka Labs

    Growing the largest community of data engineers and data scientists

  • Spotify

    Big thanks to Spotify for helping support & host NYC Data Eng!

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy