Building fast and scalable big data and ML platforms at Pinterest and JD.com

This Alluxio Meetup (www.meetup.com/Alluxio) features a chance to interact with other Alluxio (www.alluxio.io/) users and developers, as well as three talks. Thanks to our joint host Data Council (www.meetup.com/DataCouncil-AI-SF-Data-Engineering-Science)!

Agenda:
6:00pm: Happy Hour and networking
6:30pm: Tao Huang from JD.com will share “Building ad hoc and real-time data platform using Presto + Alluxio in JD”
7:00pm: Yongsheng Wu from Pinterest will share “Big data and Machine Learning at Pinterest”
7:30pm: Calvin Jia from Alluxio will share “Scalable File System Metadata Services on RocksDB, gRPC and etc: A Story from Alluxio”
8:00pm: Q&A & Mingle

Share about this!
Twitter: http://bit.ly/2WKIRzH
LinkedIn: bit.ly/2Wg6b4j

#1: How we accelerated queries by 10x using Alluxio and Presto for ad hoc and real-time stream computing at JD.

As the world’s 2nd largest e-retailer, JD.com builds the Big Data Platform (BDP) to run more than 400K jobs daily, with more than 15K cluster nodes and a total capacity of 210 PB. For our BDP, we have been running Presto on Alluxio for ad-hoc and real-time stream computing on more than 400 machines for 2+ years in production, and we have seen 10X performance gain.

In this talk I will share our BDP’s requirements, the challenges we have encountered, and how we leveraged JDPresto and Alluxio to solve those challenges. At JD, we leverage Alluxio’s HDFS compatible API and use Alluxio to connect various frameworks, including JDPresto, Spark, and Hive.

Bio: Tao Huang is a big data platform development engineer at JD.com, where he is mainly engaged in the development and maintenance of the company’s big data platform, using open source projects such as Hadoop, Spark and Alluxio.

#2: Big data and Machine Learning at Pinterest

Yongsheng, the head of big data and ML platform at Pinterest, will share his journey to build a fast and scalable big data and ML platform in AWS for Pinterest to handle the requests and complexity in data at scale. In this talk, he will cover different aspects from the requirements of the platform, the challenges encountered, the technologies chosen, and the tradeoffs that were made.

Bio:
Yongsheng Wu was one of the early engineers at Pinterest, who was instrumental in making it possible for Pinterest to scale from 10M to 300M MAUs. Yongsheng leads a team of 70+ engineers working on core online infrastructure, big data platform and ML platform to enable Pinterest quickly innovate its product and scale its revenue generation capability. Prior to Pinterest, Yongsheng worked at Twitter, Salesforce, and Oracle.

#3: Scalable File System Metadata Services on RocksDB, gRPC: A Story from Alluxio

Alluxio is an open-source distributed virtual file system that provides a single namespace federating multiple external distributed storage systems. Therefore, it is critical for Alluxio to be able to store and serve the metadata of all files and directories from all mounted external storages both at scale and at speed.

This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc. As a result of combined above techniques, Alluxio 2.0 is able to store at least 1 billion files with a significantly reduced memory requirement, serving 3000 workers and 30K clients concurrently.

Bio:
Calvin Jia is the top contributor of the Alluxio project. He has been involved as a core maintainer and release manager since the early days when the project was known as Tachyon. Calvin has a B.S. from the UC, Berkeley.