Name: [SF]Deep Dive: Spark SQL+DataFrames+Data Sources API+Parquet+Cassandra Connector
Start: 2015-10-06T18:30:00-07:00
End: 2015-10-06T21:00:00-07:00
Location: IBM Spark Technology Center

Overview

Come join us for a deep dive into the details of the spark-cassandra-connector (https://github.com/datastax/spark-cassandra-connector).

This implementation of the Spark SQL Data Sources API is one of the most advanced and performance-tunable connectors available.

Highlights of the spark-cassandra-connector

1) Token-ring aware data locality for co-location with Spark Worker nodes

2) Pushdown filter support for optimal performance and participation in the advanced Spark SQL Catalyst Query Optimizations

3) Spark 1.4, Spark 1.5 DataFrame support

4) Enables single Cassandar data store to serve both your transactional and analytics needs (pros and cons to this)

Rough Agenda

7-7:15pm: Introductions and Announcements

7:15-7:30pm: Highlights from Strata NYC

7:30-8:00pm: Spark SQL Data Sources API Overview

8:00-8:30pm: Details of the spark-cassandra-connector Data Sources API implementation

Related Links

0) Overview of the Data Sources API

https://www.youtube.com/watch?v=uxuLRiNoDio

1) The spark-cassandra-connector (https://github.com/datastax/spark-cassandra-connector) is an implementation of the Spark SQL DataSources API similar to the following:

https://github.com/databricks/spark-csv

https://github.com/databricks/spark-avro

2) Examples of the spark-cassandra-connector in action:

https://github.com/killrweather/killrweather

https://github.com/fluxcapacitor/pipeline

3) Spark SQL Data Sources API

http://blog.madhukaraphatak.com/anatomy-of-spark-dataframe-api/

http://www.river-of-bytes.com/2014/12/filtering-and-projection-in-spark-sql.html

https://github.com/spirom/LearningSpark

http://www.river-of-bytes.com/2014/12/external-data-sources-in-spark-120.html

http://www.river-of-bytes.com/2014/12/filtering-and-projection-in-spark-sql.html

Chris Fregly

AI Performance Engineering Meetup (San Francisco, Global)

Technology

Predictive Analytics

High Scalability Computing

Artificial Intelligence Programming

Artificial Intelligence Applications

Apache Spark

Data Science

Machine Learning

TensorFlow

Big Data

Deep Learning

Neural Networks

Artificial Intelligence

Natural Language Processing

Kubernetes

PyTorch

[SF]Deep Dive: Spark SQL+DataFrames+Data Sources API+Parquet+Cassandra Connector

IBM Spark Technology Center

Share

[SF]Deep Dive: Spark SQL+DataFrames+Data Sources API+Parquet+Cassandra Connector

AI Performance Engineering Meetup (San Francisco, Global)

Details

Members are also interested in