Name: Real-time file import with the vroom package (Jim Hester)
Start: 2019-09-25T18:00:00-04:00
End: 2019-09-25T19:30:00-04:00
Location: Basement Theatre of Crown Centre 1 Building

File import in R could be considered a solved problem, with multiple widely used packages (data.table, readr, and others) providing fast, robust import of common formats in addition to the functions available in base R.

However I feel there is still room for improvement in existing approaches. vroom is able to index and then query multi-Gigabyte files, including those with categorical, text and temporal data, in near real-time, parsing at over 1 Gb per second.

This is a huge boon for interactive data analysis as you can jump directly into exploratory analysis without sampling or long waits for full import.

vroom leverages the Altrep framework introduced in R 3.5 along with lazy, just-in-time parsing of the data to provide this improved latency without requiring changes to existing data manipulation code.

I will thoroughly explain the techniques used in vroom to ensure good performance, describe challenges overcome in implementing it, and provide an interactive demonstration of its capabilities.

vroom is on CRAN now, install it with `install.packages("vroom")` and learn more about the package at https://vroom.r-lib.org

Tim Hoolihan

Cleveland R User Group

R User Groups

Progressive Insurance

RStudio

Our meetup.com membership, financial support, publicity

R Consortium

Provides a bank account to manage donations

Tech Community Coalition

Provides financial support, goodies, and publicity.

Revolution Analytics

Technology

Big Data

Data Science

Machine Learning

R Project for Statistical Computing

Data Visualization

Data Analytics

Business Intelligence

Statistical Computing

Real-time file import with the vroom package (Jim Hester)

Basement Theatre of Crown Centre 1 Building

Share

Cleveland R User Group

Real-time file import with the vroom package (Jim Hester)

Cleveland R User Group

Details

Sponsors

Members are also interested in