Name: Building a Production Data Pipeline
Start: 2017-08-10T18:30:00-04:00
End: 2017-08-10T21:30:00-04:00
Location: Capital One

NYC Data Engineering Meetups are back on track! This month, we will be meeting at Capital One for an evening of talks on building a production data pipelines. Whether you are a big company or a startup, it is important to know how to build a robust pipeline that fits your use case and alerts you when things go wrong. Come and learn about the data pipelines at Button and Capital One, and hear about the challenges they face when trying to deploy their applications in production.

Schedule:

6:30 - Doors & Food
7:00 - Intro & Talk 1
7:45 - Talk 2
8:30 - Wrap & Chat

Talk 1: A production stream processing data pipeline

As data engineers we all realize the usefulness of adding stream processing to our data pipelines. Capital One has data coming in as streams from which they gather insights in real time. This talk will cover a streaming data pipeline being used at Capital One. They will walk through a use case in which they join two streams of data and cover some of the technologies in use.

You will learn:
- How companies are leveraging open source technologies to utilize real-time processing.
- About a production stream processing pipeline using Kafka, Flink and Akka
- How some companies are monitoring their production stacks

Bio:
Ali is a senior data engineer at Capital One. During his daily job, he builds APIs and production data pipelines using open source technologies. He has expertise in Python and Scala. Prior to join Capital One, he was a fellow at Insight Data Engineering. He received his bachelor and master degree in Electrical Engineering and Computer Science from MIT.

Talk 2:
Mijail Gomez from Button will be talking about the anomaly detection engine they built which alerts them when there are anomalies in their production data. He will talk about the architecture of the service - how it was designed and the challenges with the data as well as the model being used.

Bio:
Mijail is a Data Scientist at Button working on anomaly detection, data pipelines and behavioral modeling. He loves to work across the data science stack. Languages of choice are python, R, scala, and javascript. Prior to Button he worked at Mast, a telecom startup, as a software engineer. He holds a M.S in Electrical engineering and Statistics and B.S in Electrical and Computer Engineering from University of Illinois Urbana-Champaign.

Keira Zhou

NYC Data Engineering & Science (Data Council)

Technology

Software Development

Big Data

Data Analytics

Data Mining

Building a Production Data Pipeline

Capital One

Share

NYC Data Engineering & Science (Data Council)

Building a Production Data Pipeline

NYC Data Engineering & Science (Data Council)

Details

Members are also interested in