Skip to content

Scaling beyond the limits of Azure Data Factory

Photo of Richard Conway
Hosted By
Richard C.
Scaling beyond the limits of Azure Data Factory

Details

In this talk we welcome back founder Richard Conway, who will discuss a customer use case explaining how developers can overcome the shortcomings in Data Factory and Custom Activities.

Over the years ETL frameworks have struggled to offer complete compatibility with scale out patterns making design of data orchestration a very manual process with custom partitioning of data generation and transformations. Richard will describe techniques and show you to take file conversion and transformation case studies and scale out the ability of Custom Activities in a clean way to use Azure Batch using a Parametric Sweep pattern.

This talk is a must for anyone that is looking at orchestrating "Big" Data workloads with Azure Data Factory as well as those that want to understand how to pre-process data at scale for frameworks like Databricks and ADLA.

As always, this will be accompanied by beer and great company! Make sure not to miss out.

Photo of UK Azure User Group group
UK Azure User Group
See more events
WeWork 3 Waterhouse Square
3 Waterhouse Square, 138 - 142 Holborn · London