Skip to content

DoK Town Hall: Empowering developers w/ easy, scalable stream processing on K8s

Photo of Melissa Logan
Hosted By
Melissa L.
DoK Town Hall: Empowering developers w/ easy, scalable stream processing on K8s

Details

We are the Data on Kubernetes Community (DoKC), where end users go to share best practices for running data workloads on Kubernetes.

We're excited to host our monthly DoKC Town Hall virtual event in September 2024! This is an event to bring the community together to meet each other, share end-user journey stories, DoK-related projects and technologies, and keep you up-to-date on community events and ways to participate. Meetings will be held on the third Thursday of each month at 10am PT.

AGENDA
[10:00 AM]

Welcome and Community Updates
Presented by Paul Au, Head of Community

[10:05 AM]
Empowering developers with easy, scalable stream processing on Kubernetes

Speakers:
Sri Yayi, Senior Product Manager, Intuit
Vigith Maurice Principle Engineer, Intuit

Description:
Stream processing and data analytics are needed by both data engineers and non-data engineers such as platform engineers, DevOps, etc. How to make real-time stream data processing easy to use, cost-efficient and resilient to pod restarts or node upgrades is a big challenge. While there are existing stream processing solutions, they require a steep learning curve, are operationally intensive and usually costly. This talk will share our experience building a generic open-source K8s native stream processing framework called Numaflow. It enables developers to easily and quickly run large-scale stream processing jobs without needing to depend on heavy and costly data processing platforms; Using this platform, Intuit's application developers process ~5B messages for analytics, ML engineers train 135K models and make 60M predictions daily. It's been powering Intuit's large scale anomaly detection platform running in 200+ Kubernetes clusters

[10:55 AM]
DoK Quiz

Photo of Data on Kubernetes Community group
Data on Kubernetes Community
See more events