Big Data Presents... Spark in the Real World

Welcome to the first 'Big Data Partnership Presents...' meetup!

We will host a new event every quarter which focuses specifically on a variety of current and emerging technologies within the Big Data landscape.

For our first event the topic will be 'Spark in the Real World'. Our keynote speakers will deliver two presentations: Cluster Chaos and Sparking Chaos. Please find the details below:

6.00 - 6.30pm - Meet and Greet

6.30 - 6.45pm - Welcome Talk

6.45 - 7.15pm - Cluster Chaos: Resilience Testing Spark Applications

Alex Rolls (Data Test Engineer, Sky Betting & Gaming)

Have you tried switching it off and back on again? A question often asked in tech but what actually happens when you switch off critical systems in your cluster and queueing systems with data in-flight? Kafka broker down? HDFS issues in your cluster? Hive metastore down?

This talk will go through a real world example of the method used to test a real time data processing and warehousing system based on Spark and the Hadoop stack.

7.15 - 7.45pm - Pizza and Drinks

7.45 - 8.15pm - Sparking Chaos: Testing Distributed Systems

Aaron Bedward (Head of Engineering, BDP)

We are often required to develop production-grade systems that are resilient and fault-tolerant to various environmental anomalies. However, in complex distributed systems significant issues are often hidden away until after the production release. This can derail production systems and interrupt services, as well as lose data, money and Customers.

This talk will highlight some of the common problems we have faced when developing a resilient 24/7 distributed application. We will offer general tips about how to build a resilient application and demonstrate how to overcome resiliency issues by using our fault injection tool, Sparking Chaos. Sparking Chaos allows you to subject your releases to fault injection at many stages in the software design life cycle, including unit, behaviour, integration, system and scale/endurance tests.

8.15pm onwards - Drinks and Networking