Delta Lake Property Management and Choosing Deneb: A practical story
Details
Delta Lake Property Management
Managing Delta Lake table configurations, such as optimization settings (autoCompact, targetFileSize) and protocol versions, across distributed Lakehouses in Microsoft Fabric presents significant operational risks. Manual updates are prone to human error, configuration drift, and irreversible metadata corruption. This presentation introduces DeltaPropEnforcer, a robust Python/PySpark utility designed to automate the application of Delta properties with a "safety-first" architecture.
We examine the tool’s core design principles, specifically its centralized backup strategy, which stores property snapshots in a default Lakehouse to enable 100% reliable, timestamp-based rollbacks. The session covers technical challenges such as resolving OneLake GUID-based paths via SemPy to handle table renaming and special characters, implementing multi-threaded parallel processing for performance, and utilizing idempotent logic to prevent redundant operations. Furthermore, we demonstrate the tool’s safety guardrails, including "Dry-Run" previews, active write detection (skipping tables currently under modification), and the strict blacklisting of internal Delta properties to prevent corruption. This framework provides Data Engineers with a production-ready solution for maintaining consistent, optimized, and recoverable Lakehouse environments.
Choosing Deneb. A practical story of building custom visuals in Power BI.
I've been a long time using and working with Power BI and studying Microsoft Fabric and never used Deneb before, didn't have a clear business case, until last month. A manufacturing textile company makes several measurements of their products and needs to visualize the results. I couldn't show individual measurements because Power BI visuals always aggregate the data. Until I remembered Deneb and the Vega-Lite language and the solution worked magic. Let's see how.
TENTATIVE AGENDA (subject to change, all times listed are in Pacific Time):
- 4:45pm Teams Meeting opens
- 5:00pm-5:05pm Welcome
- 5:05pm-6:05pm Feature Presentation
- 6:05pm-6:10pm Wrap-up
REGISTRATION:
This meet-up will be hosted on Microsoft Teams and will be recorded for those who cannot attend live. To access the Teams Meeting link, you must RSVP for the event.
JOINING THE EVENT:
Attendees can join the Teams Meeting starting at 4:45 pm PST on the day of the event. The link will be visible only if you RSVP beforehand.
The event will begin at 5:00 pm PST. Please ensure you are on mute to minimize background noise and distractions.
During the presentation, type your questions and comments into the chat area, which will be monitored by the event hosts. There will be a Q&A session at the end, where you can unmute your mic and share your video to ask questions.