Apache Kafka and stream processing meetup @ Walmart

Join us on 22nd-Dec at Walmart Labs for Apache Kafka and stream processing meetup with talks from Walmart, Flipkart, Goibibo and MayaData.

========
Agenda
========

10:00am - 10:15am -> Registration, refreshment and introduction

10:15am - 11:05am -> Stream processing on Apache Kafka using Apache Storm & Apache Flink (Walmart)
Speaker: Karthik Deivasigamani (https://www.linkedin.com/in/karthikdeivasigamani/)

11:15am - 12:05pm -> Stream processing at Flipkart
Speaker: Arya Ketan (https://www.linkedin.com/in/aryaketan/)

12:15pm - 1:05pm -> Building a multitenant data processing and model inferencing platform with Kafka Streams (Walmart)
Speaker: Navinder Pal Singh Brar (https://www.linkedin.com/in/navinderpalsinghbrar/)

1:15pm - 1:45pm -> Lunch break

1:45pm - 2:35pm -> CDC and Kafka based data lakes (Goibibo)
Speaker: Sunny Shah (https://www.linkedin.com/in/sunny-shah-8924577/)

2:45pm - 3:05pm -> Building a Pipeline auditor using KafkaStreams (Walmart)
Speaker: Pradeep Gururaju (https://www.linkedin.com/in/pradeepgururaju/)

3:10pm - 4:00pm -> Chaos Engineering for Kafka
Speaker: Uma Mukkara (https://www.linkedin.com/in/uma-mukkara/)

4:00pm-4:30pm -> Offline discussions

======
Venue
======

Walmart Labs, Block A, First Floor Cafeteria, Salarpuria Aura building, Outer Ring Rd, Kadabeesanahalli, Bengaluru, Karnataka[masked] (https://goo.gl/maps/cc4L7eRTrUioMmLV9)

Lunch, snacks and beverages will be provided at the venue. Thanks to the Walmart team for sponsoring venue and lunch. Parking is available at the venue.

Contact: Saumitra([masked]), Navinder([masked])

=========
Abstracts
=========

Talk 1:

At Walmart we have built our catalog system (Qarth) on Apache Storm, allowing our suppliers and sellers to push their content changes to the website within minutes. Our pricing system uses Apache Flink to ensure exactly once processing and pushes price changes to the website in near real time. In this talk we will share our experience on how we leveraged Apache Storm and Apache Flink to build real time data processing system at Walmart scale. It also covers the various challenges of running Apache Flink and Apache Storm clusters in production

Talk 2:

Talk will cover
- Stream Processing use-cases and examples from Flipkart.
- Why a stream platform?
- FStream - Managed Stateful Stream Processing Platform at Flipkart.
- FStream Components

Talk 3:

Each week 275 million people shop at Walmart, generating interaction and transaction data. In this talk, Navinder provides an overview of the architecture for data processing and triggering models(processing more than 5 billion events/day), which is inbuilt for scalability and reliability. As a multitenant platform, each client’s models (such as bid models, fraud detection, and omnichannel reorder) may be interested in certain events, such as search, add to cart, transactions, etc, and whenever such an event is processed, the model interested in that particular event is triggered

Talk 4:

Talk 5:

When multiple applications are involved in handling an incoming event, there is a need to ensure that messages are not dropped un-intentionally or delayed anywhere in the flow. Pipeline Auditor that we wrote tracks if every message reaches the intended destination within the predefined SLA's. When messages are lost or miss the processing SLA, we alert on them for further action. We process ~1.5 billion events every day, at peak our stream processes 45 billion events a day

Talk 6:

Cloud Native environments are increasing at great pace primarily because of growth of adoption of Kubernetes. Kafka also being adopted in this environment. At the same time Chaos Engineering is also becoming a common practice in DevOps as a method to retain or the resilience of applications in prod. In this talk, we will give an introduction to Cloud-Native chaos engineering and how to practice the same for Kafka. Audience will get an understanding of the usage of Litmus chaos experiments for Kafka