Streaming Live Data using Apache Spark with Databricks, S3 and Kinesis
Hosted by Hybrid and Multi-Cloud Computing Meetup
Details
Streaming and Analyzing Live Data Using Apache Spark with Databricks, S3, and Kinesis
In this talk I will show you how Databricks integrates seamlessly with Kinesis and S3. I’ll introduce how to stream live data via publisher-subscribe system Kinesis to Spark running on the Databricks platform. I will show you how to join live streaming data with static data sitting on an S3. Finally, we will then analyze and display results on live visualizations running on Databricks.
In this session you will learn
- How to use Structured Streaming APIs
- Perform Spark SQL queries with live and static data
- Understand an end-to-end continuous application
++
Dorothy Kucar works in the Technical Training group at Databricks, where she writes code using advanced Databricks/Spark features. Previously she’s been a Data Engineer at various AdTech and FinTech companies. She started her career at the IBM TJ Watson Research lab developing algorithms for circuit layout.