StreamSets at 2U, and the StreamSets Test Framework

Join us for this first StreamSets New York City User Group Meetup!

Agenda:

6pm - Food and beverages, and a chance to mingle and get to know each other.

6:30pm - StreamSets at 2U

Oleg Berfirer (https://www.linkedin.com/in/olegberfirer/) is Director of Data Systems at 2U, where he is building a financial data warehouse and the analytics around it.

Oleg will share his experiences of building dataflow pipelines with StreamSets Data Collector to read data from sources such as Amazon Kinesis, Rest API end points, Elasticsearch and PostgreSQL, transforming data in the pipeline, and writing into destinations such as Amazon RedShift, S3, SQS, Salesforce and Neo4j.

7:15pm - STFU: A Short Course on the StreamSets Test Framework

Dima Spivak (https://www.linkedin.com/in/dmitryspivak/) is a Director of Engineering at StreamSets, where he leads the Engineering Productivity team. Before joining StreamSets, he was a software engineer at Cloudera. Dima is also a committer and PMC member on the Apache HBase project.

One of the biggest challenges faced by engineers implementing DataOps solutions is testing them before going into production. At StreamSets, where the development of such solutions is our key focus, functional and integration testing presented a unique challenge. In this talk, Dima will describe the solution his team developed and are open sourcing, the StreamSets Test Framework. He'll also share some of the important lessons he learned along the way, and provide some insights for how his work could help you in solving your own DataOps challenges.

8pm - Close