Scaling beyond the limits of Azure Data Factory

In this talk we welcome back founder Richard Conway, who will discuss a customer use case explaining how developers can overcome the shortcomings in Data Factory and Custom Activities.

Over the years ETL frameworks have struggled to offer complete compatibility with scale out patterns making design of data orchestration a very manual process with custom partitioning of data generation and transformations. Richard will describe techniques and show you to take file conversion and transformation case studies and scale out the ability of Custom Activities in a clean way to use Azure Batch using a Parametric Sweep pattern.

This talk is a must for anyone that is looking at orchestrating "Big" Data workloads with Azure Data Factory as well as those that want to understand how to pre-process data at scale for frameworks like Databricks and ADLA.

As always, this will be accompanied by beer and great company! Make sure not to miss out.