Design Framework for Big Data Computing

cnidaria: A Generative Communication Framework for Distributed Computing

Presentation by Dr Michael Kane from Yale University

Abstract:

This presentation will introduce a scalable, distributed software design that facilitates communications patterns beyond those supported by existing MapReduce frameworks making it appropriate for a more general class of computing challenges.

Unlike existing framework (e.g. MPI) this design is elastic and supports operations for mixed sparse and dense data representations, including numerical computing. This generality is achieved through a generative communication scheme that provides transparent data movement in a peer-to-peer fashion.

The design integrates advances in both "cloud" computing and distributed numerical computing and can be used to implement scalable computing solutions.