Please join us to learn more about the Spice AI and Apache Spark™ Connect Use Case for Operational AI 🤝
Agenda:
- Welcome and Introductions
- Talk1: Extending Apache Spark for Operational AI with Spice AI and Spark Connect
- Q & A
Talk Title: Extending Apache Spark for Operational AI with Spice AI and Spark Connect
Abstract: Apache Spark is the de facto choice for large-scale data processing. For operational use cases requiring low-latency, high-concurrency data access, data-intensive and AI applications often require additional infrastructure. Spice AI OSS and Spark Connect, a single-node data and AI compute engine built in Rust, bridges this gap by integrating with Apache Spark Connect to enable federated SQL queries across Spark clusters and operational data systems, delivering sub-second query performance for operational workloads.
This talk details how Spice AI OSS extends Spark’s capabilities, leveraging Spark Connect’s decoupled architecture for scalable, secure data access and integrating with Mosaic AI models for real-time decision-making. Attendees will learn the technical motivation for integrating with Spark Connect.
A live demo will showcase. Spice AI querying a Spark-hosted dataset, achieving sub-second query times, and feeding results into a Mosaic AI model for real-time operations
What Attendees Will Learn
- Spark Connect Architecture: Understand Spark Connect’s client-application model, gRPC protocol, and its role in decoupling compute and client applications for operational use cases.
- Spice AI OSS Integration: Learn how Spice AI OSS configures Spark Connect (spark_remote: sc://:15002) for federated SQL queries, including dataset definitions in spicepod.yaml and secure authentication.
- Operational Use Cases: Explore how Spice AI extends Spark to support low-latency, high-concurrency queries for data-intensive and AI applications, unifying data access across cloud, edge, and on-premises environments.
- Mosaic AI Integration: Discover how Spice AI leverages Mosaic AI model serving to process Spark query results for real-time decision-making, such as predictive analytics or recommendation systems.
Bio: Luke Kim is the Founder and CEO of Spice AI, VC-backed startup focused on data and AI infrastructure for intelligent applications. Spice AI is the creator of Spice AI Open Source, a portal compute runtime that unifies federated data query, retrieval, and AI inference to accelerate mission-critical, operational workloads. Luke previously led engineering teams at Microsoft, most recently as the co-creator of the Azure Incubations group in the Azure CTOs Office. A vocal advocate for open-source, he drives Spice AI’s contributions across the data and AI ecosystem, including Apache Arrow, Apache DataFusion, and Apache Spark.
📅 Date: July 30 2025
⏰ Time: 9:30 AM - 10:50 AM PST
📍 Location: online
