The Data Must Flow

Speaker: Simon Hewitt
Talk: prefect.io - The Data Must Flow

This is a sneak peak of a talk Simon will give at DataKind later in the month on his experiences with https://www.prefect.io

It's extremely rare that data sits in one place, or in one state for long. Inevitably it must be gathered, manipulated, augmented, validated, anonymised and a million other things before we can begin to extract any value from it.

In the same way, that processing usually starts with a script, that gets enhanced and built on top of, and refactored, with lines commented out and distributed across everyone's laptop. If we're lucky it might be in source control...

Prefect takes your code and transforms it into a robust, distributed pipeline, taking raw input in one end and reliably transforming into usable data at the other.

Starting with a high level overview, we'll follow the evolution from a one file python script through to a reliable pipeline, available to your whole team.