Designing an Execution Framework for Azure Data Factory w/ Andy Leonard

Session category: General
Track: Analytics
Topics – Analytics: ETL, Microsoft Azure Data Factory
Level: 300 (Experienced)
Title: Designing an Execution Framework for Azure Data Factory
Abstract:
Attend “Designing an Execution Framework for Azure Data Factory” and takeaway:

1. Comparison and contrast of Azure Data Factory debug and triggered execution.
2. Benefits and limitations of the Execute Pipeline activity.
3. Benefits and complexity of using the Azure Data Factory REST API for execution.

Experience or familiarity with developing, scheduling, and monitoring Azure Data Factory pipeline executions is suggested.
If your enterprise manages dozens of ADF pipelines, an execution framework is unnecessary overhead. If your enterprise manages thousands of ADF pipelines? You need an execution framework. But how should you implement it? Should you take a straightforward approach and use the Execute Pipeline activity or implement a metadata-driven approach using the Azure Data Factory REST API?
Prerequisites: Experience or familiarity with developing, scheduling, and monitoring Azure Data Factory pipeline executions is suggested.

Goals:

1. Comparison and contrast of Azure Data Factory debug and triggered execution.
2. Benefits and limitations of the Execute Pipeline activity.

Benefits and complexity of using the Azure Data Factory REST API for execution.

About Andy:
Andy Leonard is a husband, dad, and grandfather; creator of – and Data Philosopher at – DILM (Data Integration Lifecycle Management) Suite; a blogger, founder and Chief Data Engineer at Enterprise Data & Analytics; an SSIS and Azure Data Factory trainer, consultant, and developer; a SQL Server database and data warehouse developer; an author, mentor, engineer, and farmer.