Large-scale text processing pipeline with Spark ML and GraphFrames

6:00 PM- 6:30 PM: drinks, mingling

6:30 PM - 8:30PM: Large-scale text processing pipeline with Spark ML and GraphFrames with Alexey Svyatkovskiy, Ph.D. (Big Data scientist at Princeton University)

https://www.linkedin.com/in/asvyatko/

In this talk we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark ML and GraphFrames. Histogrammar package—a cross-platform suite of data aggregation primitives for making histograms, calculating descriptive statistics and plotting in Scala—is introduced to enable interactive data analysis in Spark REPL. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale. In the hands-on portion of the meetup session I will show how to write custom Spark ML transformers given a UDF, will dig deeper into pre-­processing and feature extraction and explain how to tune number of buckets in HashingTF, will discuss possible approaches to all­-pairs similarity calculation.