PySpark vs SparkSQL for a Presentation Layer

Hari Gnanaprakasam, software engineer at Capital One, will walk us through his analysis of PySpark and SparkSQL. He compared these two approaches for data access to a presentation layer he needed to create for Capital One analysts. Learn…

• How Spark was used to create a presentation layer on top of S3

• Differences between PySpark and SparkSQL in syntax, time-to-market, resource utilization, and performance

Hari will also demo the comparison.

____________________________________________

Hari is a polyglot engineer at Capital One. He has...

• enabled Acceptance Test Driven Development(ATDD) for more than 100+ teams,

• developed frameworks to automate large scale Data migration between Teradata and Hadoop.

• developed a concurrent loader for Snowflake data migration, data sharing between Snowflake and Spark, and built and deployed Apache Spark based orchestration solutions.

____________________________________________

Directions:

From 288 South, take the Capital One Drive exit.

From 288 North, take the West Creek Parkway West exit and then your first left onto Capital One Drive.

Follow the signs to the "Commons" - that's not where we're meeting but it will take you past Central Parking. Park in the Central Parking deck.

When you exit the deck, you'll be facing two buildings. Enter the building on the right, in the doors that face the parking deck - this is the Town Center.

After checking-in with security (bring an ID), go up the stairs. We will be congregating at the top of the stairs for something to eat before moving to room 24A for the presentation.