Johannesburg Data Platform meeting 13 May 2025

Agenda

18:30 - Welcome and introduction

18:45 - Michael Johnson: Delta Merge, the data engineer`s best friend

The ‘UPSERT pattern’, where a set of data changes is combined with existing data, is a pattern commonly used in data engineering. The UPSERT pattern allows you, the data engineer, to merge INSERTS, UPDATES and DELETES. Often it is only possible to perform these steps as separate operations which can be both time-consuming and error prone.

Delta Merge was added to Delta Lake to simplify the UPSERT process for data engineers, streamlining the process into a single command that handles the inserts, updates and deletes as a single operation.

During this session you will learn how you can use the PySpark or SparkSQL to seamlessly merge change data sets efficiently to implement common data modeling techniques such as Type 1 or Type 2 dimensions or soft deletes all or which are commonly used in data warehousing scenarios.

Join us to find out how the Delta Merge statement can really become the data engineer's best friend, saving you time to focus on what matters.

20:00 - Closing and prize giving