Auckland Data Eng meetup, March 2025

Hey all, please join us for our March Auckland Data Eng meetup. Join us for an engaging session filled with exciting discussions and networking opportunities.

Don't miss out—RSVP to secure your spot. A big shout-out to Plexure for providing us with the venue and Catering! 🎉

🏠 Location: 2 Graham Street, Auckland Central, Auckland 1010
🍕 Catering: Databricks 😊
💬 Join our Slack Group here: Data Engineers Slack Channel

Schedule:
- 5.00pm: Doors Open
- 5.30pm: Announcements and Welcome
- 5.40pm: 1st Talk (25 mins + 5 min Q/A)
- 6.10pm: 2nd Talk (25 mins + 5 min Q/A)
- 6.40pm: Networking
- 7.30pm: Doors Close

Speakers:
🎤 Vicky Avison, Principal Data Engineer, Plexure
Talk Title: Metadata-Driven Streaming Ingestion with Delta Live Tables, Azure Event Hubs & Azure Schema Registry
Talk Summary: At Plexure, we ingest hundreds of millions of customer activities and transactions into our data platform every day, fuelling our personalisation engine and providing insights into the effectiveness of marketing campaigns.We’re on a journey to transition from infrequent batch ingestion to near real-time streaming using Azure Event Hubs and Delta Live Tables. This transformation will allow us to react to customer behaviour as it happens, rather than hours or even days later.It also enables us to move faster in other ways. By leveraging a Schema Registry, we’ve created a metadata-driven framework that allows data producers to:
- Evolve schemas with confidence, ensuring downstream processes continue running smoothly.
- Seamlessly publish new datasets into the data platform without requiring Data Engineering assistance.In this talk, I'll talk about our journey and discuss how we’re implementing this with DLT meta-programming — including a live demo of the end-to-end process.
Speaker Bio: Vicky is the Principal Data Engineer at Plexure, where she drives the technical vision and engineering standards for the data platform. She stumbled into the world of Big Data over 10 years ago and has been building and optimising data platforms and pipelines ever since.

🎤 Steven MacLeod, Founder at Occam Software
Talk Title: The future of data is convergence + the force multiplier
Talk Summary:
There have been two significant ideas in the past year that should lead to a seismic shift in how we think about data. Joe Reis' talk at the DataEngBytes conference on 'The future of data is convergence' and Chris Tab talking about 'The force multiplier'
This talk will combine the two ideas and provide some real-world examples pointing to the future of data
Speaker Bio: I have spent 30,000 hours analysing why systems fail, have time delays and cost overruns. And another 30,000 hours working out what to do about it.

If you'd like to present at future meetups, please submit your talk here!

Remember to bring along some great questions! See you all there...