Past Meetup

Tracking commercial aircraft using a Raspberry Pi, Kafka, Vertica & Dataiku

This Meetup is past

214 people went

Location image of event venue


1. This event is at Big Data London @ Olympia, London W14 8UX
2. Registration for this event is FREE, but MUST be done via the Big Data London website:
3. Registration will also give you access to Big Data LDN for the TWO days (13-14 Nov)


18:00 Tracking commercial aircraft in near real-time using a Raspberry Pi, Kafka and Vertica
Mark Whalley

18:45 Anomaly detection in aviation
Silviu Tofan

19:30 Networking

Tracking commercial aircraft in real-time using a Raspberry Pi, Kafka and Vertica

Through 2017, the Big Data & Machine Learning (London) Meetup group held a series of presentations on how, using relatively simple to acquire hardware and software, it is possible to track the flights of commercial aircraft in near real-time.
Automatic Dependent Surveillance Broadcast (ADS-B) data from aircraft transponders is captured and decoded using a Raspberry Pi, passed through a series of Apache Kafka Topics before being loaded into a Vertica database. Subsequent presentations went on to show how this streaming real-time, combined with historic data could be manipulated and used with relative ease using the in-built analytics capabilities of Vertica for data capture, enrichment, measuring and preparing. Demonstrating simple SQL functions for handling time series data, gap filling and interpolation, sessionization, outlier detection etc.
The perfect IoT use case with thousands of aircraft (“devices”) providing 10’s millions of messages per day, all of which can be captured, decoded, loaded and analysed in seconds.
The first presentation by Mark Whalley, Vertica Systems Engineer, will summarise the whole end-to-end project. From introducing ADS-B, building the Raspberry Pi, digital broadcast receiver and decode/ETL software, through to integration with Apache Kafka, the Vertica Kafka Scheduler, ingestion into Vertica and then on to performing simple visualisations through to data preparation and Machine Learning.
Mark will also bring along one of the functioning Raspberry Pi setups for those interesting in seeing the rig up close.
At the outset of this project, a challenge was set for anyone to get involved in this project, and to take what had been started to the next level.

The second presentation at this event by Silviu Tofan from Dataiku does just this.

Anomaly detection in aviation

Making use of Series Flight Tracking project, Silviu will look at a practical use case of flight data using anomaly detection. In this scenario, working together with Vertica, Dataiku were provided flight tracking data across a region in Europe, near a major airport. Going first through exploratory steps, we will walk you through the definition of our anomaly detection case, how this is achieved using time series data, as well what are the best way of putting such a model into production.

Mark Whalley
From the early 1980s, Mark worked with Michael Stonebraker's Ingres RDBMS and then a number of column-store big data analytic technologies. In 2016, he joined HPE Big Data Platform as a Systems Engineer specializing in Vertica, Vertica SQL in Hadoop and more recently with Vertica EON. In September 2017 he followed Vertica as it merged into Micro Focus – one of the world’s largest pure-play software companies in the world.
Mark frequently delivers talks at the London, Cambridge and Munich Big Data & Machine Learning Meetups, the British Computer Society - Advanced Programming Specialist Group, Vertica Forums and elsewhere. He is a technical blogger and author.

Silviu Tofan
Silviu is a data scientist at Dataiku. Coming from a business-oriented background, with an MSc in Business Analytics from the University of Manchester, he transitioned to a more technical focus while working on an optimization problem together with ARM. Before coming to Dataiku, Silviu was involved with social housing organizations to develop their data science capabilities.