Design Patterns for Cloud Based Data Lakes / Implications for next generation BI

Abstract:
With data management increasingly moving to object storage and cloud data warehouses, organizations naturally expect their BI applications to also benefit from the scale of data and real-time analytics.
However, traditional BI architecture in this cloud-native environment faces some not-so-obvious challenges: While object storage scales to store large amounts of data, analytics can only be done on subsets.
And while data can be ingested in real time, you have to wait for data warehouse batch ETL cycles before doing analytics. Object stores in the cloud provide interesting new challenges and require new design patterns to address those challenges.

It’s time to invest in an agile cloud-native architecture for BI that doesn’t rely on outdated analytic patterns of traditional BI.

Shant Hovsepian explains how the application of traditional BI to cloud-native data management and warehousing can result in a “frankenstack”—the exact issue that many users have tried to simplify in the first place by choosing cloud. Shant then offers an overview of considerations for service-oriented design of the clouds (storage, compute, catalog, security, SQL service, etc.) and discusses why BI should be a native service in this environment.

Shant Hovsepian is a cofounder and CTO of Arcadia Data, where he is responsible for the company’s long-term innovation and technical direction. Previously, Shant was an early member of the engineering team at Teradata, which he joined through the acquisition of Aster Data. Shant interned at Google, where he worked on optimizing the AdWords database. His experience includes everything from Linux kernel programming and database optimization to visualization. He started his first lemonade stand at the age of four and ran a small IT consulting business in high school. Shant studied computer science at UCLA, where he had publications in top-tier computer systems conferences.