May Meetup: Designing for Speed: Spark SQL in Fabric (Jean Joseph)

Designing for Speed: Mastering Spark SQL Joins, Partitions, and Execution Plans

Description: Is your Spark workload crawling when it should be flying? Poor design choices like inefficient partitioning, bloated file sizes, and underutilized compute can quietly cripple performance in Microsoft Fabric. This session is a must-attend for data engineers and architects who want to uncover the hidden costs of bad design and learn how to unlock Spark’s full potential.

We’ll explore how Microsoft Fabric’s Spark environments empower you to harness host-level compute power with precision. From selecting optimized Spark runtimes to configuring resource profiles tailored for read-heavy or write-heavy workloads, Fabric gives you granular control over memory, cores, and session behavior. You’ll learn how to leverage starter pools for rapid session startup or build custom Spark pools with autoscaling and dynamic allocation to match your job’s complexity.

Attendees will also demystify the Spark Catalog and the Catalyst Optimizer, understanding how logical plans are transformed into efficient execution strategies and how schema design and query syntax can make or break optimization. We’ll dive into advanced techniques like V-Ordering, Delta compaction, and partition pruning, showing how they reduce I/O and accelerate query performance.

Whether you're troubleshooting sluggish pipelines or architecting for scale, this session delivers actionable insights to help you design smarter, run faster, and make Spark in Fabric work for you not against you.

Teams Meeting Link: https://teams.microsoft.com/meet/28463863990050?p=cKrHLyiXi6CMLOAWxp