Building robust, scalable, and adaptive applications on Spark Streaming

6:30-7pm Mingling
7:15-7:45pm Technical Talk
7:45-8pm Q&A
8pm Mingling

Join us to see Tathagata Das will present his Spark Summit East talk.

Abstract:
As the adoption of Spark Streaming increases rapidly, the community has been asking for greater robustness and scalability from Spark Streaming applications in a wider range of operating environments. To fulfill these demands, we have steadily added a number of features in Spark Streaming. We have added backpressure mechanisms which allows Spark Streaming to dynamically adapt to changes in incoming data rates, and maintain stability of the application. In addition, we are extending Spark’s Dynamic Allocation to Spark Streaming, so that streaming applications can elastically scale based on processing requirements. In my talk, I am going to explore these mechanisms and explain how developers can write robust, scalable and adaptive streaming applications using them.