OpenLineage Hybrid Meetup

Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift without a shared standard for lineage metadata. It requires duplication of effort across pipeline tooling, and deployment of new tools can break existing lineage workflows. Getting useful lineage can seem like a sisyphean task.

Enter OpenLineage, an increasingly adopted open standard for lineage metadata collection. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.

Join us at the SF Astronomer offices & on Zoom on September 12th at 6:00 pm PT to learn more about the OpenLineage spec and integrations. You’ll meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a robust discussion about the future of the project.

Agenda:

• Unlocking Data Products with OpenLineage at Astronomer: Julian LaNeve and Jason Ma, Astronomer.
• OpenLineage: From Operators to Hooks by Maciej Obuchowski, Astronomer+GetInData/Xebia.
• Activating Operational Metadata with Airflow, Atlan and Openlineage by Kacper Muda, GetInData/Xebia.
• Hamilton, a Scaffold for all Your Python Platform Concerns (and a New OpenLineage Producer) by Stefan Krawczyk
• Lightning Talk on New Marquez Features and the Marquez Project Roadmap by Willy Lulciuc, Marquez Lead, and Peter Hicks, Marquez Committer.

Thank you to our sponsors:
Astronomer
GetInData/Xebia
LFAI & Data Foundation

Someone in the lobby will direct you to the seventh floor. If you arrive late, send us a message in Slack.

Zoom link to be provided here.