Intro to Building a Distributed Pipeline for Real Time Analysis of Uber's Data

Introduction to Building a Distributed Machine Learning Pipeline for Real Time Analysis of Uber Data Using Apache APIs: Kafka, Spark, and HBase

In this talk we will look at a solution that combines real-time data streams with machine learning to analyze and visualize popular Uber trip locations in New York City. You will see the end-to-end process required to build this application using Apache APIs for Kafka, Spark, and HBase.

According to Gartner, by 2020, smart cities will be using about 1.39 billion connected cars, IoT sensors and devices. The analysis of behavior patterns within cities will allow optimization of traffic, better planning decisions, and smarter advertising. You may be excited about the possibilities of exploiting data streams to gain actionable insights from continuously produced data in real-time but you may find it difficult to conceptualize how to implement such a solution. We will walk you through an architecture that combines data streaming with machine learning to enhance Uber trip data to analyze and visualize the most popular pick-up/drop-off locations by date and time so that drivers’ locations could be optimized and priced according to demand. The presentation will consist of four sections:

• Introduction to Spark machine learning for developers

• Kafka and Spark Streaming

• Real time dashboard using a micro service framework

• Using the Spark HBase connector for parallel writes and reads

About the speaker:

Carol McDonald is a solutions architect at MapR focusing on big data, Apache Kafka, Apache HBase, Apache Drill, Apache Spark, and machine learning in healthcare, finance, and telecom. Previously, Carol worked as a Technology Evangelist for Sun, an architect/developer on: a large health information exchange, a large loan application for a leading bank, pharmaceutical applications for Roche, telecom applications for HP, messaging applications for IBM, and sigint applications for the NSA. Carol holds an MS in computer science from the University of Tennessee and a BS in geology from Vanderbilt University.

About our host venue:

Honeywell creates some of the world’s most sophisticated software-based technologies that play a major role in the Internet of Things (IoT), helping everything from aircraft, cars, homes and buildings, manufacturing plants, supply chains, and workers become more connected to make our world smarter, safer, and more sustainable. If you want to make a difference in these critical industries, you can apply here – https://www.honeywell.com/careers