Coping with x2,000 data volumes growth

I worked in a startup and our volumes started small, really small. I was the data scientist but I needed to actually build a data platform first. By the end of 6 months, daily volumes were 2,000 times the size. This talk takes you through my 6 months of learning how to build a high-scale data platform whilst requirements, scale, and technology were all changing. We’ll look at the various bits of Azure I used, how they started off or became wrong for my situation, how I designed the system to cope with the constant swapping and changing of technologies, and then we’ll look at the system I ended up with that could cope with the scale and at then some.

About Steph Locke
Steph Locke leads a life of data, coffee, books and board games. During the day, Steph runs her own data consultancy helping people get better with data. Steph enjoys being her own gal as it means she gets to spend plenty of time building communities to provide platforms for people to help each other be better with data.

Steph runs two user groups in Cardiff for Microsoft technologies and R, whilst supporting others to help her local area learn more. She used to help organise SQL Relay each year to help folks all around the country get access to free learning. She’s now working on a global series of conferences for R called satRdays, and a local conference Battle of the Beards. Steph blogs and presents, whenever she can, to share her own knowledge and experience. In recognition of her efforts, Microsoft continue to award her the Microsoft Data Platform award for her contributions in the SQL and R worlds.