Optimising Data Pipelines

This event is organised through Den Norske Dataforening - BI and Analytics Group (www.dnd.no).

The Data Engineering Oslo Group has been quiet for some time due to covid so let us make this group active again with 'Optimising Data Pipelines' talk. The event will be held at Rebel, Universitetsgata 2, Oslo on 26th January. Questions and discussions are very welcome!

In the field of data engineering, a data pipeline is a set of processes that move data from one place to another. Smart data pipelines are an important part of this process because they automate many of the tasks involved in moving and transforming data. This allows data engineers to focus on other important tasks, such as designing and building systems that can extract value from the data.

Agenda:
17:00 - Meet&Greet
17:10 - Den Norske Dataforening - Introduction
17:15 - Saleem Pothiwala - Optimising Data Pipelines
18:00 - Food break
18:30 - StreamSets - Data Integration for the Cloud World
19:15 - Networking & Mingling
20:00 - Closing

One important aspect of Data Engineering is the ability to detect and manage data drift, which is the term used to describe changes in the characteristics of a dataset over time. Smart data pipelines can help with this by providing alerts and notifications when they detect changes in the data that may indicate data drift while ingesting the data without any intervention.

We will discuss the scenarios and see some live demos of smart data pipelines.

3 key takeaways

• Smart data pipelines can automate many of the tasks involved in moving and transforming data, allowing data engineers to focus on other important tasks.
• Smart data pipelines can provide alerts and notifications when they detect data drift, allowing data engineers to quickly address any issues.
• Smart data pipelines can be configured to automatically take corrective action when data drift is detected, ensuring that the data remains accurate and consistent.

StreamSets is a Software AG company that believe in the audacious, ambitious goal of teasing order out of the chaos of modern data. StreamSets helps its customers achieve that goal by ensuring data engineering teams thrive in today’s world of constant change. StreamSets brings enterprise-proven DataOps capabilities to modern data integration, enabling continuous data for the modern data stack.

Really looking forward to the talks, and seeing you all again! Please RSVP sooner as we have limited number of seats.