August 2018 Meetup

For the August meetup, our friends at Zendesk are kindly hosting us, again!

Agenda:

7pm: Meet & Greet

7:30pm: "Introduction to Apache Airflow – Programmatically Manage Your Workflows for Data Engineering" by Xiaodong Deng

Apache Airflow was started in 2014 at Airbnb, then became an Apache incubator project in 2016.

Mainly built with Python, Airflow is a platform to programmatically author, schedule and monitor workflows. Workflows can be defined as directed acyclic graphs (DAG) and configured as Python scripts. It supports complex task dependency management, distributed task execution (like Celery + RabbitMQ), and integration with different technology stacks/tools (RDBMS, AWS, Google Cloud, HDFS, Spark, etc.). Friendly UI is also provided for monitoring and configuration.

Currently Airflow is adopted by many companies, including Airbnb, Glassdoor, Google, HBO, ING, Quora, Yahoo!, and Zendesk. Alibaba also built a workflow scheduling system named "Maat" on top of Airflow.

Xiaodong DENG is a data engineer at DBS Bank. He had experience working on data science & engineering in research sector and insurance industry. Trained in mathematics background, Xiaodong believes that solid engineering implementation is the key to converting theory to reality (http://XD-DENG.com).

8pm: "Dota Flask - A Dota 2 recommendation engine written in Flask" by Wang Eric GaoXiang

In this talk, we will explore how to build a recommendation engine using OpenDota API and Python Flask. We will also discuss about best practice in writing Python Flask application as well as how to use Docker with Python Flask. At last, Eric will share the project roadmap for an open source project with the audience, and if you are interested in helping the project, we can discuss more about it.

Eric started programming with Python at RP & NUS. He joined HPE Singapore Lab as a full stack developer and gained exposure to microservices, devops and big data technology. He worked at DBS as a data engineer and today is with Silot.ai as a senior software developer to explore fintech and blockchain solutions.