Past Meetup

Scaling beyond the limits of Azure Data Factory

This Meetup is past

75 people went

WeWork 138 Holborn

138 Holborn · London

How to find us

Right by Chancery Lane station, enter square and building is on the corner, wait at reception and we'll come and get you.

Location image of event venue

Details

In this talk we welcome back founder Richard Conway, who will discuss a customer use case explaining how developers can overcome the shortcomings in Data Factory and Custom Activities.

Over the years ETL frameworks have struggled to offer complete compatibility with scale out patterns making design of data orchestration a very manual process with custom partitioning of data generation and transformations. Richard will describe techniques and show you to take file conversion and transformation case studies and scale out the ability of Custom Activities in a clean way to use Azure Batch using a Parametric Sweep pattern.

This talk is a must for anyone that is looking at orchestrating "Big" Data workloads with Azure Data Factory as well as those that want to understand how to pre-process data at scale for frameworks like Databricks and ADLA.

As always, this will be accompanied by beer and great company! Make sure not to miss out.