Redis Streams + Spark Structured Streaming: Scale-out Your Continuous Apps

Please sign up on Skillsmatter ( https://skillsmatter.com/meetups/12377-redis-streams-spark-structured-streaming-scale-out-your-continuous-apps) so they know how many people to expect.

Title: Redis Streams + Spark Structured Streaming: A Perfect Combination to Scale-out Your Continuous Applications

Speaker: Roshan Kumar, Sr. Product Manager, Redis Labs (visiting from Silicon Valley)

Agenda:
7:00 pm Registration, Appetizers & Networking
7:15 pm Welcome & Introductions
7:30 pm "Redis Modules" Lightning Talk by Pieter Cailliau of Redis Labs
7:45 pm "Redis Streams + Spark Structured Streaming" by Roshan Kumar of Redis Labs
8:45 Q&A

Talk Description: "Redis Streams + Spark Structured Streaming"

Continuous applications face many challenges as they grow to production. Often, due to the rapid increase in the number of devices or end-users or other data sources, the size of their data set grows exponentially. This results in a backlog of data to be processed. The data will no longer be processed in near-real-time.

These continuous applications have 3 things in common:

1. Collect data from sources (ex: IoT devices)
2. Process data in real-time (example: ETL)
3. Deliver data to machine learning serving layer for decision making

Redis Streams enables you to collect both binary and text data in the time series format. The consumer groups of Redis Stream help you match the data processing rate of your continuous application with the rate of data arrival from various sources.

Apache Spark’s Structured Streaming API enables real-time decision making for Continuous Applications.

In this session, Roshan will perform a live demonstration of how to integrate open source Redis Streams with Apache Spark’s Structured Streaming API using Spark-Redis library. He will also walk through the code and demo a live continuous application.

About Roshan:

Roshan Kumar is a Senior Product Manager at Redis Labs. He has extensive experience in software development and product management. In the past, Roshan has worked at Hewlett-Packard, and Silicon Valley startups – ZillionTV, Salorix, Alopa, and ActiveVideo to name a few. As an enthusiastic programmer, he designed and developed mindzeal.com, an online platform hosting computer programming courses for young students. Roshan holds a Bachelor’s degree in Computer Science, and MBA from Santa Clara University.