Name: New Functions and Workflow Examples with Spark DataFrames
Start: 2016-01-18T18:00:00-08:00
End: 2016-01-18T20:00:00-08:00
Location: "The Aquarium" at Plenty of Fish

The use of DataFrames in Spark (introduced in Spark 1.3) is becoming more powerful with every subsequent release. While the use of RDDs is still a very effective way of processing data in Spark, the use of DataFrames confers advantages such as improved readability and faster processing. The functions work very well in pipelines and are easy to reason about since they provide better semantic information about data transformations. The speed benefit arises from DataFrame operations internally representing logical plans. Before execution, such plans are optimized by Catalyst and transformed into physical plans. Is it likely that DataFrames will become the primary API for most users, so it’s now a great time to dive into processing data with DataFrames in Spark.

Rich Iannone will provide an overview of many of the functions introduced from Spark versions 1.3 to 1.6. Because examples are a great way to quickly gain understanding, I’ll provide plenty of those. These will be practical examples that are easily applicable to a variety of data transformation tasks. There will be a live-coding session to see Spark DataFrames used in step-by-step analyses. This will provide an opportunity for Q&As and other participation amongst the Meetup members. I’m really excited to show you all the great new additions to the DataFrame API so I hope to see you all at the session!

George Chow

Rich Iannone

Vancouver Apache Spark Meetup

Technology

Python

Scala

Hadoop

Big Data

MapReduce

Machine Learning

Data Analytics

Data Mining

Data Science

Distributed Systems

Apache Spark

New Functions and Workflow Examples with Spark DataFrames

"The Aquarium" at Plenty of Fish

Share

Vancouver Apache Spark Meetup

New Functions and Workflow Examples with Spark DataFrames

Vancouver Apache Spark Meetup

Details

Members are also interested in