OpenLineage Meetup @ Google

Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift without a shared standard for lineage metadata. It requires duplication of effort across pipeline tooling, and deployment of new tools can break existing lineage workflows. Getting useful lineage can seem like a sisyphean task.

Enter OpenLineage, an increasingly adopted open standard for lineage metadata collection. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.

Agenda:

• Mary Idamkina: OpenLineage in GCP Dataplex
• Paweł Leszczynski: Updates on the Spark Integration
• Jakub Dardziński: "Extracting lineage from PythonOperator - how come this is possible?"
• Paweł Leszczynski: "How to become spark-openlineage contributor in 5 steps"