Ibis: A Journey in Distributed Computation w/ Ryan Spangler

Recently we at Little Bird had a series of issues trying to get another Clojure distributed computation library working in production. Learning from the frustrations involved in that process clarified exactly what we wanted out of a distributed computation system, and soon after Ibis was born. The goal was to simplify everything and leave the hard problems of synchronization to battle tested libraries that already do it well, namely Zookeeper and Kafka. Built on this solid foundation, Ibis was a natural expression of our intent when designing a distributed system. Providing seamless distribution in the face of adding and removing nodes on the fly, streaming, scheduling and unique tasks, Ibis is already serving us well in production and we consider it a great success.

We have open sourced the code and you can find the repository, along with a guide of how to use Ibis here: https://github.com/littlebird/ibis

In this talk I will go through the design decisions, the journey of implementation (in particular how Ibis uses Kafka and Zookeeper), a dive into the code and philosophical ramblings as they may arise. Hope to see you there!

BIO - Ryan Spangler is Lead Developer at Little Bird, a local startup here in Portland. He works on a wide variety of things including graph algorithms, network analysis, data visualization, machine learning and distributed systems. In his spare time he is obsessed with biological problems, designing board games and making strange tones with a variety of musical instruments.