Data Engineer's Lunch: Building a Data Lakehouse on your Laptop Workshop

Join Anant and Alex Merced from Dremio in this hands-on workshop where participants will embark on a journey to construct their very own data lakehouse platform using their laptops. The workshop is designed to introduce and guide through the setup and utilization of three pivotal tools in the data lakehouse architecture: Dremio, Nessie, and Apache Iceberg. Each of these tools plays a crucial role in enabling the flexibility of data lakes with the efficiency and ease of use of data warehouses, aiming to simplify and economize data management.

Participants will start by setting up a Docker environment to run all necessary services, including a notebook server, Nessie for catalog tracking with Git-like versioning, Minio as an S3-compatible storage layer, and Dremio as the core lakehouse platform. The workshop will provide a practical, step-by-step guide to federating data sources, organizing and documenting data, and performing queries with Dremio; tracking table changes and branching with Nessie; and creating, querying, and managing Apache Iceberg tables for an ACID-compliant data lakehouse.

Prerequisites for the workshop include having Docker installed on your laptop. Attendees will be taken through the process of creating a docker-compose file to spin up the required services, configuring Dremio to connect with Nessie and Minio, and finally, executing SQL queries to manipulate and query data within their lakehouse.

This immersive session aims not just to educate but to empower attendees with the knowledge and tools needed to experiment with and implement their data lakehouse solutions. By the end of the workshop, participants will have a functional data lakehouse environment on their laptops, enabling them to explore further and apply what they have learned to real-world scenarios. Whether you're looking to improve your data management strategies or curious about the data lakehouse architecture, this workshop will provide a solid foundation and practical experience.