Building data pipelines with Scala

Record Union will talk about a newly developed data pipeline that has been written from scratch in Scala, to handle and analyse music data collected by the company. The pipeline is using two gnarly Scala technologies, Akka streams and Apache Spark, and is being deployed on Google Cloud Platform using, among others, the managed services DataProc and GKE. We will go into some detail about the architecture but also discuss how it is to try to write purely functional Scala code while navigating among Scala/Java idiosyncrasies, actors and distributed systems. Some deployment and testing strategies might show up as well.

Agenda:

• 18:00 Doors open

• 18:30 Presentations begin

• 19:30 Mingle

• 20:30 Doors close

Food and beverages (Falafel with vegetarian and meat options) will be available.

The entrance door should be open from 18:00, otherwise ring Contract Analytics at the entrance door and go to the top floor or call 0709713705 if you're late.