Past Meetup

Developing a Fast and Durable Pub/Sub Message Bus

This Meetup is past

75 people went

Location image of event venue

Details

Speaker: Will Sewell (https://www.meetup.com/London-Haskell/members/65994472/).

Over the past year, Pusher (http://pusher.com/) has been using Haskell to write a new message bus for their core publish and subscribe messaging system.

Customers currently send more than 50,000 messages per second through the existing system and they aim to keep latency below 50ms. This means hard requirements on the performance of the new system. On top of this, durability guarantees in the face of failovers and client disconnections should be provided.

Will describes their experience with using Haskell for this problem.

On the one hand, they found Haskell's type system and testing capabilities excellent for writing reliable production software. As an example QuickCheck was used test their raft implementation (https://blog.pusher.com/fuzz-testing-distributed-systems-with-quickcheck/).

On the other hand, they encountered challenges when attempting to optimise for high throughput and low latency. The most fundamental of these problems was the length of GC pauses when there was a large working set in memory (https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/).

In the end they ended up switching to using Go (https://golang.org/), and Will also covers their experience with making the transition.