Decodering: Turning the table on big data

Redshift and Snowflake have made column-oriented databases everyday tools for processing big data. But what does column-oriented actually mean, and why would anyone store their data sideways?

A friendly chat with an old friend left me realising I've got some gaps in my understanding. For my own edification, I've researched column-oriented databases and how they've developed, relating it to my experience working with the teams who operate and use them.

I spent years as product owner of a very large data pipeline. We started with a proprietary column store, Hadoop, and a whole mess of custom Java, with a daily fight to get the data in and the answers out. Then we moved the lot to AWS, and I got a close-up look at the same ideas growing up into Redshift, Parquet files in S3 and, eventually, Snowflake.

Along the way: how a column store lays out and compresses its data, why sort order matters so much, what you give up, and where the idea turns up now, from cloud warehouses to DuckDB on a laptop.

Decodering is a place for informal talks. This is not a lecture, so please interrupt whenever you like. If you've got your own stories or lessons, do bring them.

Expect at least one anecdote that stretches too far.