Data Engineering Use Cases : Airflow, Cost , and BQ ML

Details

Agenda:
1800 - 1830 Networking
1830 - 1900 "Data Engineering Use Cases : Airflow, Cost Reduction , and BQ-ML" , Omid Vahdaty, Big Data Ninja
1900 - "Observability and Monitoring for Data Engineering" Evgeny Shulman,CTO and a co-founder of databand.ai.

what will you learn in "Observability and Monitoring for Data Engineering"

Apache Airflow as a golden standard of Data Engineering, what does it have already for observability, what’s missing? We will take a look at what’s need to be observed in data-centric products, how observability helps with CI/CD, and how can we make our day to day job more productive.

Evgeny Shulman is a CTO and a co-founder of databand.ai. He was a chief architect and a founding employee of Crosswise (acquired by Oracle), creating one of the most advanced working environments for Machine Learning and Data processing. He served in an elite 8200 department (cyber) and holds a B.Sc in CS from Technion.
databand.ai is a software platform for accelerating the Data/ML development lifecycle.

Target audience: Data Engineers, Data Science Practiatials who do MLops by themself, Apache Airflow users or similar systems.

What will you learn in "Data Engineering Use Cases : Airflow, Cost , and BQ-ML"
In this lecture I will share with you how I solved DE use cases using Airflow, BQ, and BQ-ML having Architecture, Cost, Performance in mind.

1. Data Transfer Use Case - BigQuery 93% Cost Reduction
2. Similarweb API data pipeline | Airflow & Cost reduction
3. K Means Via BQ ML Demystified