Options for Ingest: Elasticsearch Ingest Node and Apache Airflow

There are many options for big data ingest - at this meetup we'll learn about Elasticsearch (https://www.elastic.co/products/elasticsearch) Ingest Nodes and the Apache Airflow (incubating) (https://wiki.apache.org/incubator/AirflowProposal) project.

This event is hosted by Elastic (https://www.elastic.co/) and sponsored by StreamSets (http://streamsets.com/).

6 - 6:30 pm - Food and networking.

6:30 - 7:15 pm - Tal Levy (https://www.linkedin.com/in/talevy), Software Engineer at Elastic (https://www.elastic.co/), "Ingest Node: Enriching Documents within Elasticsearch"

When ingesting data into Elasticsearch, sometimes only simple transforms need to be performed on the data prior to indexing. Enter Ingest Node: a new node type that will allow you to do just that! This talk will introduce you to Ingest Node and how to integrate it with the rest of the Elastic Stack.

Tal Levy is a Logstash (https://www.elastic.co/products/logstash) developer based in Mountain View, CA. Tal joined Elasticsearch from Quixey (https://www.quixey.com/) where he worked on data systems built with Kafka and Hadoop. When he is not click-clacking on keyboards, you can find him cycling and climbing.

7:15 - 8 pm - Vijay Bhat (https://www.linkedin.com/in/vijaysbhat) - Data Science Consultant, "Fuss Free ETL - Tame Your Pipelines With Apache Airflow"

Data scientists spend a lot of time doing (and redoing) tedious ETL work. This is all the more so if they don't have data engineers to support their ETL pipelines. It doesn't have to be that way. This talk will cover Airflow (https://github.com/apache/incubator-airflow), an awesome open source ETL workflow tool developed by Airbnb (and inspired by Facebook's Dataswarm ETL system). We will go over how data scientists can setup, monitor and self-service their pipelines without data engineering's support. There will also be a live demo.

Vijay Bhat is a data scientist in San Francisco, currently consulting at Capital One. He brings industry insights from his work with companies in the smart meter analytics, adtech, fintech and social media domains.

Vijay's blog is at http://vijaybhat.com (http://vijaybhat.com/)