Spark Meetup - Real-Time Edition

Hi All,

we made it: We pulled together another Spark Meetup.

This time Corvil is going to host us.

As usual we will open the doors at 6:30pm and will have some pizza and beers to support the mingling.

Talks will start at 7:00pm.

We first have Richard and Luka talk about Real-Time Big Data Analytics for Financial Trading with Spark.

Terry (new co-organizer) is currently working to line up two more quick lightening talks after that.

This looks like a great evening to meet some of your old friends from the Spark community or ... to make a couple new ones :).

Hope to see you there.

Regards ...
Roland

Title: Real-Time Big Data Analytics for Financial Trading with Spark

Presenter: Richard Gleeson (Director, Product Management - Pico) and Luka Boskovic (Principal Big Data Software Developer - Pico).

Bio: With more than 15 years of experience in product management for IT Operations Analytics at Micromuse, IBM and Corvil, Richard Gleeson now directs Pico’s real-time big data analytics product for financial trading – Intelligence Hub.
Luka Boskovic has been a lead software developer working on Big Data solutions for Corvil and Pico for 5 years, and a principal contributor to Intelligence Hub’s architecture.

Target Audience: Data professionals with an interest in applications of Spark eco-system technologies to real-world problems, particularly in finance.

Abstract: Modern financial trading is a fully automated high-speed environment in which humans are too slow to keep track of what’s going on. In this presentation we will talk about how we built a real-time analytics platform based on Spark to help users understand how their trading systems are performing. We will explain the nature of data available in trading environments and how it is used for performance analysis, the architecture of our solution and the analytics that we run on it, and the remaining unsolved challenges that we still face. Our talk will describe the experiences we have had using Spark with a variety of eco-system technologies including Druid, Presto, Superset, Kafka, Ignite, MemSQL and ClickHouse.

Alongside our main talk, we will be having two shorter "lightening talks". These talks are designed so people who may be unsure about speaking first time in the group, or want to talk about a specific point that may not be enough for a main talk.

Lightning Talks

#1 - Introduction to join skew in Spark
Daniel Mateus Pires, Software Engineer at Earnest Research will be bringing us through join skew in spark.

#2 - Test Driven Development with PySpark
Andrew Farrell, Senior Data Engineer at Tenable will be taking us through TDD using Pyspark