The Evolving Landscape of Data Engineering & How Systems Fail

Welcome to our first edition of the Data Engineering Club, hosted by Twitter!

In order to attend, you must RSVP @ https://twitterdataeng.splashthat.com/

Agenda:

6:30 - 7:00pm: Doors open. Networking, eat, drink, mingle!

7:00 - 7:30pm: The Evolving Landscape of Data Engineering (Andrei Savu)

7:30 - 8:15pm: How Systems Fail (Alexander Huras)

8:15 - 9:00pm: Hang out

Details on Talks and Speakers:

"The Evolving Landscape of Data Engineering"

Abstract: Data Engineering is a relatively new, but fast evolving discipline that spans multiple environments and technologies, from traditional data centers to hyper-scale cloud providers, a discipline that combines closed-source, homegrown and open source software to create scalable data pipelines and power incredible new product features.

In this presentation, we will go over the last 5-10 years of technology trends and advancements and bring all of that together in a story about modern day Data Engineering and the magic behind it.

About Andrei Savu:

Andrei is a software engineer in the MoPub team. He got the "data-bug" in college while developing an application to explore EXIF metadata from a very large collection of photos stored by Adobe. That led to multiple open source contributions (Apache Whirr, Zookeeper etc.) and a startup (Axemblr.com) that was acquired by Cloudera to bootstrap the Cloud Engineering team.

As a Tech Lead at Cloudera, he had the opportunity to drive and witness first-hand many of the recent developments in the field of Data Engineering and Advanced Analytics and as of now he applies that knowledge to advance mobile monetization at Twitter part of MoPub.

As a co-organizer of the Data Engineering Club, he is excited to learn about the latest challenges faced by Data Engineers in the Bay Area and beyond.

"How Systems Fail"

Abstract: With modern tools, building and maintaining data pipelines are as easy and simple as describing your system's process graph. There is no concept of "deployment", and "versioning" is just something you read about in old textbooks---as if things weren't always correctly built the first time. Somewhere, a phone rings.

You wake up, it's 4:00 am and you're on call. Your phone has been ringing for the last 15 minutes and likely escalated the page to the secondary on-call: also you. Great. You wipe the sleep from your eyes, backflip out of bed, and quickly get into the VPN. It looks like a couple queue subscribers are lagging---you thought the team had fixed that problem hope that it isn't another case of data corruption...

When we talk about cool data technology, we rarely mention what happens when things go wrong, when the abstractions start leaking, when product requirements change or appear out of nowhere. This will not be one of those talks.

About Alexander Huras:

Alex is a systems engineer on Twitter's Revenue-Platform team. He's the tech lead for the performance forecasting work group, and has designed and implemented multiple mission-critical "Big Data" systems. Before that, Alex worked on Traffic network optimization, brain simulations, and guitar soloing. Alex is a functional programming evangelist and is currently working on the next generation of Twitter's analytics products.

Welcome to the club!