Crowdmix - An Event Based Social Music Platform & Kafka 0.10 New Features

6.30pm - Doors open, Food + Drinks, Network

7.00pm - Talk - "Music Streams - Running a social network on an event based architecture" by Stefano Galarraga, Michal Dziemianko and Tiago Palma from Crowdmix

Designing a social network offers some exciting challenges to engineers. The system needs to operate at scale, to provide a responsive user experience and to be able to inspect user activity in order to both generate new content and improve how the existing content is delivered.

Event Driven Architectures are particularly suitable to handle these kind of challenges and highly scalable messaging systems such as Apache Kafka have been designed specifically to support the requirements of modern high volume applications.

In this talk we are describing how the Crowdmix back-end has been designed as an Event Based system running on top of Kafka. We are going to present the overall system architecture and discuss in more detail some of the different sub components processing those events in different fashions, from streaming based processing to batch processing passing through a lambda-style batch and stream cooperation.

We are going to conclude describing some lessons learned from our one-year journey in implementing and operating the system

Speaker Profiles

Stefano Galarraga is currently working as Lead Developer at Crowdmix. Started his professional career in 1997 and has been working mostly in middleware, and message based systems, most recently moving to Big Data. He is contributor of Twitter’s Scalding and Typesafe’s AKKA project plus some of his owns you can find at https://github.com/galarragas

Michal Dziemianko is currently working as Big Data Engineer and Data Scientist at Crowdmix. He has a background in AI and Distributed Computing. He has PhD in Machine Learning from the University of Edinburgh and has been working for around 10 years in software engineering and research.

Tiago Palma is currently working as Big Data Engineer and Data Warehouse Developer at Crowdmix. He has several years of experience in Data Warehousing and ETL, working with traditional MPP systems as Teradata and with Big Data technologies. He has also experience as a DevOps.

7:45pm - Break

8:00pm - Talk - "Whats new in Kafka 0.10" by Eno Thereska, from Confluent

Apache Kafka 0.10 is a major release and this talk will go over some of the new features and enhancements.

In particular, one of the main contributions is Kafka Streams, a library that turns Apache Kafka into a modern stream processing systems. Kafka Streams includes a high level language for describing common stream operations (such as joining, filtering, and aggregating records), allowing developers to quickly develop powerful streaming applications.

Speaker Profile

Eno is a software engineer at Confluent. Prior to Confluent, Eno was a senior researcher at Microsoft Research where he led the incubation and tech-transfer of several projects related to file and storage systems. He received his PhD from Carnegie Mellon University.

8:40pm - Flash Talk - "A look at the new sink connector for streaming Kafka messages to PostgreSQL (and compatible databases)" by Duncan Pauly, CTO of JustOne (www.justonedb.com)

Flash Talk includes set up and configuration details, performance expectations and example use case.