Skip to content

Details

Come join us at 6:00 PM for food, beer, and scalable in-memory file systems - with the original co-creator of the Tachyon project - Haoyuan Li !

Memory is the key to fast Big Data processing. And the Tachyon project (http://tachyon-project.org/) has emerged as a powerful framework for storing large amounts of data scalably in memory for light-speed processing with Spark and other HCFS compatible tools, at scale.

The importance of in-memory caching has been realized by many, and frameworks such as Spark and Shark, which are already leverage memory performance.

As data sets continue to grow, storage is increasingly becoming a critical bottleneck in many workloads. To address this need, we have developed Tachyon (http://tachyon-project.org/), a memory centric fault-tolerant distributed file system, which enables reliable file sharing at memory-speed across cluster frameworks, such as Spark and MapReduce.

Tachyon (http://tachyon-project.org/) is Hadoop compatible. Existing Spark and MapReduce programs can run on top of it without any code changes. Tachyon is the default off-heap option in Spark. The project is open source and is already deployed at multiple companies. In addition, Tachyon has more than 60 contributors (https://github.com/amplab/tachyon/graphs/contributors) from over 15 institutions, including Yahoo, Intel, Redhat, and Pivotal. The project is the storage layer of the Berkeley Data Analytics Stack (BDAS (https://amplab.cs.berkeley.edu/software/)) and also part of the Fedora distribution (http://timothysc.github.io/blog/2014/02/17/bdas-tachyon/).

In this meetup, Haoyuan Li, the co-creator of the Tachyon project, will give a overview of the project, including motivation, current status, and its roadmap.

Related topics

You may also like