November Sydney Data Engineering Meetup

Campaign Monitor have been gracious enough to host us in their offices this month.

1st Speaker - Pranavi Chandramohan, Campaign Monitor
Navigating the Stream
With increased demand from businesses and customers to provide real time analytics and reporting, the need for stream processing has become inevitable.
At Campaign Monitor, we have started to move from batch processing towards a stream processing model to provide real time reports to our customers. There are many ways to implement stream processing out there; in this talk we will take a look at Kafka streams, Spark structured streaming and Spark Streaming.

2nd Speaker - Deependra Shekhawat, AWS
Apache Airflow On AWS - Lessons Learned. In this talk we will share key best practices that we have seen data engineers adopt while deploying apache airflow on AWS. We will specifically discuss how we used AWS ECS (Fargate) to deploy a production Airflow setup using the CeleryExecutor, the lessons we learnt while operating the environment and share some key best practices while deploying and operating an Airflow installation.
Deependra Shekhawat is a Techincal Account Manager (TAM) at AWS Sydney. In his role as TAM he has been working very closely with AWS Enterprise Support Customers in Australia and specifically with data engineering teams to help build production grade data processing pipelines using open source software and AWS services.

3rd Speaker - Mike Seddon, AGL
Introducing Arc (https://aglenergy.github.io/arc/), an opinionated framework for defining predictable, repeatable and manageable data transformation pipelines.
Mike is a Senior Data Engineer at AGL Energy and has co-developed the Arc framework.