Event: Streaming things with Kafka and Spark

18:00 - 18:30 - Mingling
18:30 - 19:30 - Stream, Stream, Stream: Different Streaming methods with Spark and Kafka - Itai Yaffe and Ron Tevel @ Nielsen (Hebrew)
19:30 - 20:15 - Nielsen presents: Fun with Kafka, Spark and offset management - Simona Meriam @ Nielsen (English)

PARKING

There is a free 3 hours parking in TLV Fashion mall (5 minutes walk from the office) and free parking at Givon parking for Discount bank card holders.

Title: Stream, Stream, Stream: Different Streaming methods with Spark and Kafka

Abstract:
Going into different streaming methods, we will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty...).
We will also present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services’ costs.
Topics include :

• Kafka and Spark Streaming for stateless and stateful use-cases
• Spark Structured Streaming as a possible alternative
• Combining Spark Streaming with batch ETLs
• “Streaming” over Data Lake using Kafka

Bio:
Itai Yaffe is a Big Data Tech Lead at NMC, dealing with Big Data challenges for the past 6 years.

Ron Tevel is a Big Data Developer at NMC, developing Big Data infrastructure solutions.

Title: Nielsen presents: Fun with Kafka, Spark and offset management - Simona Meriam @ Nielsen (English)

Abstract:
We’ll start our talk by explaining how we used to manage our Kafka consumer offsets against Spark-Kafka 0.8 consumer.
Next, we’ll review the problems we encountered during the upgrade to Spark-Kafka 0.10 consumer.
We’ll finish by going into the depths of the solution we ended up implementing.

Bio:
Simona Meriam is a Big Data engineer at NMC, focused on designing Big Data infrastructures and trips to Japan.