Spark in the NHS and Cloud Object Stores

It's been a little while since our last meetup but we are now back with a new venue and two great talks from the recent QCon and Spark Summit conferences. There will be the usual beer and pizza to help move things along too.

Looking forward to seeing you there!

Challenging Perceptions of NHS IT

Speaker: Dan Rathbone with Ed Hiley

Abstract:

What are your perceptions of NHS IT? Not great? Well the truth is very different to what you might expect. There is something of a technical renaissance going on in parts of the NHS where things are being done in a modern way, learning from past experiences.

We'll look at one example system where we've built a highly resilient multi-data centre data processing system utilising techniques more often used in the cloud but with bare metal servers. See how we’ve built automated performance tests, an immutable infrastructure and a NoSQL data store with support for versioning data.

Bio: Edward Hiley is a Principal Systems Engineer with NHS Digital. Since joining NHS Digital, Edward has worked on national service such as the Secondary Uses Service (SUS) replacement project: SUS+. SUS+ is a "ground up" full replacement of the current application that involves myriad challenges, including immutable infrastructure, disputed compute clusters, and multi data centre. Prior to joining NHS Digital, Edward was a solution architect for the Health and Social Care Information Centre and an Associate Director for the National Institute for Health and Clinical excellence.

Dan Rathbone is co-founder and Technical Director of Infinity Works, a 100-strong consultancy and software house based out of Leeds and London. Over the years Dan has held many varied roles focussing on areas from infrastructure to front end development and most things in between. Drawing on a broad skill set Dan now builds and operates high-scale and high-performance systems for Infinity Works’ clients. Most recently Dan has been working with NHS Digital to drive the modernisation of their critical national services, re-engineering them using FOSS, end-to-end DevOps teams and Agile and Lean delivery techniques.

Spark and Object Stores —What You Need To Know

Speaker: Steve Loughran

Abstract: If you are running Apache Spark in cloud environments, Object Stores —such as Amazon S3 or Azure WASB— are a core part of your system. What you can’t do is treat them like “just another filesystem” —do that and things will, eventually, go horribly wrong. This talk looks at the object stores in the cloud infrastructures, including underlying architectures, compares them to what a “real filesystem” is expected to do and shows how to use object stores efficiently and safely as sources of and destinations of data. It goes into depth on recent “S3a” work, showing how including improvements in performance, security, functionality and measurement —and demonstrating how to use make best use of it from a spark application. If you are planning to deploy Spark in cloud, or doing so today: this is information you need to understand. The performance of you code and integrity of your data depends on it.

Bio: Steve Loughran works at Hortonworks on leading-edge Hadoop applications, most recently in high-performance Amazon's S3 storage support in Hadoop and Spark, as well as long-lived Yarn Services

He's the author of Ant in Action, a member of the Apache Software Foundation, and a committer on the Hadoop core since 2009. Prior to joining Hortonworks in 2012, he was a Research Scientist at HP Laboratories.

He lives and works in Bristol, England. For fun he falls off bicycles in the local woodland.

twitter: @steveloughran

slides: http://slideshare.net/steve_l