Building a Distributed File System for the Cloud-Native Era
Details
This will be a hybrid event. Attendees can join both in-person as well as online. To attend online please register here: https://www.alluxio.io/events/meetup-bdb/
Today, data engineering in modern enterprises has become increasingly more complex and resource-consuming, particularly because (1) the rich amount of organizational data is often distributed across data centers, cloud regions, or even cloud providers, and (2) the complexity of the big data stack has been quickly increasing over the past few years with an explosion in big-data analytics and machine-learning engines (like MapReduce, Hive, Spark, Presto, Tensorflow, PyTorch to name a few).
To address these challenges, it is critical to provide a single and logical namespace to federate different storage services, on-prem or cloud-native, to abstract away the data heterogeneity, while providing data locality to improve the computation performance. [Bin Fan] will share his observation and lessons learned in designing, architecting, and implementing such a system – Alluxio open-source project — since 2015.
Alluxio originated from UC Berkeley AMPLab (used to be called Tachyon) and was initially proposed as a daemon service to enable Spark to share RDDs across jobs for performance and fault tolerance. Today, it has become a general-purpose, high-performance, and highly available distributed file system to provide generic data service to abstract away complexity in data and I/O. Many companies and organizations today like Uber, Meta, Tencent, Tiktok, Shopee are using Alluxio in production, as a building block in their data platform to create a data abstraction and access layer. We will talk about the journey of this open source project, especially in its design challenges in tiered metadata storage (based on RocksDB), embedded state-replicate machine (based on RAFT) for HA, and evolution in RPC framework (based on gRPC) and etc.
SPEAKER: Bin Fan, VP Open Source and Founding Engineer, Alluxio
SPEAKER BIO: Bin is the PMC maintainer of Alluxio open source. Prior to joining Alluxio as a founding engineer, he worked for Google to build the next-generation storage infrastructure. Bin received his Ph.D. in Computer Science from Carnegie Mellon University on the design and implementation of distributed systems.
Big Data Bellevue Meetup was created by Intelius and takes place on the Eastside. Intelius provides the only centralized service for delivering comprehensive information about people, places, organizations, and their connection to each other. Our state-of-the-art big data technology platform is utilized across a wide range of industries to implement specific solutions.
On the third Thursday of each month, we invite an industry leader in Big Data to give a presentation followed by a lively discussion on big data technology and its impact on business world. Past speakers include researchers from the University of Washington, as well as senior members of various companies, such as Microsoft, Amazon, eBay, IBM, MapR and inome.
Microsoft graciously provides free drink and refreshments!


