Webinar: Optimizing Data Latency and Query Latency in Open Table Formats

To attend, register here.

Data lakes came with an implicit trade. You got cheap storage and open formats, and in exchange you accepted that the data would be hours old and the queries would take minutes. Low cost, high latency. That trade is no longer necessary. Streaming data continuously into open table formats and running indexed, page-level query execution over those same files turns the lake from a batch archive into a near-real-time serving layer. Low cost and low latency, on one copy of the data, in object storage.

This session walks the path an event takes from producer to end user. Confluent Tableflow represents Kafka topics and their schemas directly as Apache Iceberg and Delta Lake tables, with schematization, schema evolution, and catalog publishing handled for you, so there is no separate ingestion stack to build and no batch window to wait on. StarTree, built on Apache Pinot, then queries those same open tables with page-level precision, using indexes and pruning to touch a small fraction of the Parquet files a scan-based engine would read. The result is a single copy of governed data that serves both exploratory analysis and the sub-second, high-concurrency queries that user-facing applications and AI agents require.

We will cover the architecture, show it running on a live stream, and share the latency numbers behind each stage.

What You Will Learn

• How to go from Kafka topic to queryable Iceberg table without custom pipelines
• What page-level indexing changes about the economics of querying open table formats
• Which application and AI use cases this architecture enables

Who Should Attend

• Data platform leaders and senior data engineers responsible for Kafka, Iceberg, Delta Lake, or analytical serving
• Architects evaluating how to serve low-latency queries without duplicating lakehouse data
• Application and AI platform teams that need fresh governed data under high concurrency