Skip to content

Portland Azure User Group Virtual Meetup

Photo of Gregory Petrossian
Hosted By
Gregory P.
Portland Azure User Group Virtual Meetup

Details

Metadata Driven Pipelines for Dynamic Full and Incremental Processing in Azure SQL

Developing ETLs/ELTs can be a complex process when you add in business logic, large amounts of data, and the high volume of table data that needs to be moved from source to target. This is especially true in analytical workloads involving Azure SQL when there is a need to either fully reload a table or incrementally update a table. In order to handle the logic to incrementally update a table or fully reload a table in Azure SQL (or Azure Synapse), we will need to create the following assets:
• Metadata table in Azure SQL
◦ This will contain the configurations needed to load each table end to end
• Metadata driven pipelines
◦ Parent and child pipeline templates that will orchestrate and execute the ETL/ELT end to end
• Custom SQL logic for incremental processing
◦ Dynamic SQL to perform the delete and insert based on criteria the user provides in the metadata table

This session uses Azure SQL DB as the source and sink databases. However, Azure SQL MI, On-Prem SQL, and Synapse Dedicated Pools (along with Synapse Pipelines) will also be compatible for this solution. As a source, you can use databases like MySQL, Oracle, and others. You will just need to adjust the query syntax/connections to match the desired source.

Photo of Portland Azure User Group group
Portland Azure User Group
See more events