Improving the Write Scalability of the CernVM File System with Erlang/OTP

In conjunction with Code Mesh 2017 (http://www.codemesh.io/codemesh2017) (7-9 November), we'll have Code Mesh speaker Radu Popescu from CERN speaking at the next meetup!

[Get an exclusive 10% off Code Mesh tickets with our code ESLmeetup10]

Guest speaker: Radu Popescu

Radu Popescu is currently a CERN fellow, working on the CernVM-FS distributed file system project, a mission critical system of the CERN physics collaboration. He really enjoys distributed systems, expressive type systems and functional programming. He believes studying programming language design makes for a better programmer, regardless of the tools used.

Radu has worked on a multitude of other interesting projects, including a 3D editor for photogrammetry data - Pix4Dmapper (https://pix4d.com/product/pix4dmapper-pro) and libraries for high performance computing (HPC) - Trilinos (https://github.com/trilinos/trilinos), LifeV (https://github.com/lifev/lifev). His relationship with HPC culminated in a PhD degree obtained at the Ecole Polytechnique Federale Lausanne, Switzerland.

Talk title: Improving the Write Scalability of the CernVM File System with Erlang/OTP

The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service. It was developed at CERN to assist High Energy Physics (HEP) collaborations in deploying software on the worldwide distributed computing infrastructure for data processing applications, but it can also be used in other domains.

CernVM-FS is implemented as a POSIX read-only file system in user space (FUSE) and uses content-addressed storage and Merkle trees for maintaining file data and meta-data. Files are stored remotely on standard web servers and are retrieved and cached on-demand through outgoing HTTP connections only, avoiding most of the firewall issues of other network file systems.

For writing, CernVM-FS follows a publish-subscribe pattern with a single source of new content that is propagated to a large number of readers.

This talk focuses on a new publication architecture, capable of accommodating dynamic strategies for content publishing and improving the scalability of the publication phase. We chose Erlang/OTP for implementing certain components of the new architecture, due to its suitability for making scalable and fault-tolerant systems.