Data Engineering on AWS by Thorsten Greiner

We are very happy to have Thorsten Greiner for a second talk, this time about his current work!

7:00pm Socializing

7:15pm Better IDEAs!

Large organizations face multiple challenges related to data: how to make data that is stored across separate silos available for analysis? How to give the business tools to handle data and gain meaningful insights? How to scale up with new business and new demands?

This talk gives an overview of how the IDEA team at Intuit solves these problems.

As a central part of this endeavor, we have built an Entity Store using Cassandra and Kafka. This system handles several thousand transactions per second and stores tens of TB of data. The talk will outline the design decisions and summarizes the experiences gained in developing and operating this system.

Looking forward, the talks features the challenges that come with bringing this system into the public cloud - e.g. how to handle security, manage infrastructure and migrate from an on-premise data center.

Thorsten's Bio

Working with Java since 1997. Focussing on Big Data since 2014. Engineering Manager with Intuit Data Engineering & Analytics since April 2016. With his team he drives real time big data solutions for analytics challenges and deploys them in the public cloud.