Big Data and Optics with Julia

We WILL have 🍕pizza and will be at Harvard this time! Join us at Palfrey House on the Harvard campus.

Talk 1: Accessors.jl beyond @set, or a tour of the opticland
Speaker: Alexander Plavin, Black Hole Initiative

The Accessors.jl package is reasonably well-known as a way to update values within immutable structs with its @set macro. However, the underlying concept of "optics" is much more powerful and widely applicable.

The Accessors design makes these optics impressively seamless and performant in Julia, heavily relying on the multiple dispatch for composability. In the talk, I'll touch upon the design and implementation, and outline several neat usecases – from autodiff and function optimization to tabular operations and plotting.

Talk 2: Answering localized questions on big datasets with RangeExtractor.jl
Speaker: Anshul Singhvi, JuliaHub

Our world today is defined by big data; the output of a single satellite orbit is larger than your laptop's hard drive. The canonical way to analyze "big earth observation datasets" has always been to throw it on a cluster and let it run overnight. But what if it didn't have to be?

RangeExtractor.jl is a Julia package that is meant to accelerate localized computations over global, large (1TB to 1PB), and chunked datasets. It does this through multithreading, asynchronous data downloads, and intelligent query-splitting over chunk boundaries. In the talk, I'll touch over the case for RangeExtractor, how it's designed, and what performance bottlenecks remain to be improved upon.

I'll also talk about the two principal usecases that motivated RangeExtractor: (a) extracting the lowest elevation of each glacier for all ~200,000 glaciers in the world, over a 30-meter resolution elevation grid, and (b) computing elevation statistics over the whole of Greenland. Both of these questions will be answered on my 14-inch Macbook, without pre-downloading any big data.