How To Design for Big Data

A successful product will often see a growth in the data it stores and processes.

The downside of this success is that data storage can grow beyond the limits of a single traditional database, and data processing can outgrow the memory limits of a single computer.

We're very lucky to have someone as experienced as Rob Harrison to talk us through:

1. The common problems that emerge as your data outgrows your application.

2. The key design principles for architecture that can scale beyond a single-computer database and memory.

3. Examples of technology and designs that have proven themselves to scale - including at CERN.

Rob will explore:

• Designing for success - architecting redundant applications which continue to perform under load.

• Parallelism and programming paradigms - writing applications that allow for both fast iteration and optimisation.

• Benefits of microservices - how to perform distributed processing of large data sets across clusters.

Whether you're starting small, starting big, or growing an existing service, this talk will provide valuable advice for successfully scaling your products and services - advice based on Rob's practical experience of what works and what doesn't.

---

We will also have two short flash talks:

1. John Moore explaining his work on AI generated game narratives, and inviting participation in his project.

2. Ian Mason from Exeter University on the Smartline for-social-good research project exploring sensor data from homes in Cornwall, and an invitation to participate in data mining hackathons.

---

We will also have a flash talk on using AI to develop narratives and storylines for games.