No More Drama: Just Clean Data - Tillmann Eitelberg (oh22)

Hi, Data Folks from Hamburg!

Here is the invitation for November 2024. We are planning an exciting evening about one of the building blocks of each Database: Data Quality.

Attention!
We gained another great room sponsor: ERGON Datenprojekte! Don't worry; you won't have to figure out a complete new route for the evening, as the venue is virtually on the opposite street side of the Main Central Station Hamburg, in the city center.

And Tillmann promised that he would take care of the culinary delights. Thank you!

• 18:00 Open door and warm up networking round
• 18:15 Intro
• 18:30 No More Drama: Just Clean Data
• 20:00 Open discussion about the topic
• 21:00 The End...

No More Drama: Just Clean Data
Data quality issues like inconsistent formats, missing values, and inaccurate data can severely impact decision-making. Most of us have spent hours chasing down these errors, and as data pipelines grow in complexity, validation and cleansing alone aren’t enough. Data testing and observability are crucial to catch and prevent these issues early.
However, integrating data quality solutions presents its own challenges—tools can be too technical for business users or not flexible enough for engineers, and sometimes data needs to be moved to external SaaS environments, making integration difficult.
Our Data Quality Solution HEDDA.IO addresses these challenges. It’s a cloud-native solution that integrates easily into processes without altering data flows. Supporting pySpark on Databricks, Microsoft Fabric, and Azure Synapse, it also offers a C# library as well as REST services, Azure Functions and container app jobs.
For business users, HEDDA.IO includes a user-friendly interface for managing data and discussing business rules, along with a new Excel Add-In that allows teams to analyze data in familiar environments, ensuring collaboration across departments.
In this session, I’ll explore common data quality challenges, and explain how processes like testing and observability can improve solutions. I'll also discuss available integrated tools within platforms and demonstrate how HEDDA.IO simplifies solving these problems while engaging all stakeholders. Lastly, I’ll share insights on how we designed HEDDA.IO to seamlessly integrate with Azure and other tools.