A Data Engineer trying to do something actually useful with GenAI

As a Data Engineer, you're likely familiar with the buzz around Generative AI (GenAI) and its potential to revolutionize data processing and analysis. But how can you actually leverage GenAI to drive tangible business value? Here is a fellow data engineer trying to unlock practical, on-the-job value with Generative AI in our field (This goes beyond replacing Stackoverflow and google with ChatGPT)

This talk and the demos are based on open or local LLMs and using your regular dev tools. The session aims to get the audience familiar with general concepts on GenAI, followed by application of these concepts and frameworks for data operations. In the end we will talk about the big question - how to (and do have to) actually productionise data systems using Gen AI.

Talk Outline:

Introduction to LLMs (10 minutes)

• Overview of all the concepts and buzzwords from a fellow lay-engineer
• Suggestions (opinionated) on the best tools and frameworks for local development

Data Operations / Day in the life of a DE (5 minutes)

• Data acquisition
• Data modelling
• Reporting
• Building Data pipelines
• Quality check on data and pipelines
• Governance

How can LLMs help (20 minutes)

• Generating synthetic data
• SQL and structuring
• Visualizations
• Analysis
• Quality checks and testing

Challenges and future work (5)

• New(er) kids on the block
• Building real world products - tests, scale and monitoring
• Are we as DE replaceable by AI?