Near Real-Time Retrieval-Augmented Generation (RAG) pipelines at Scale in Java

​We use the Luma site to host our events. SIGN UP HERE

RAG is the pattern used to improve LLM accuracy and minimize AI hallucinations. But most implementations are a patchwork of batch embedding jobs, external vector databases, message brokers, miscellaneous services, and brittle glue code. The result is predictable: higher latency, tougher and more costly operations.
In this talk, you’ll see a streaming RAG architecture built natively in Java: continuous ingestion and transformation with distributed DAG pipelines, horizontally scalable embedding inference, and in-memory distributed vector collections for millisecond semantic search. We’ll show how data can be vectorized as it arrives from CDC, events, REST sources, and documents, how partition-aware processing cuts network overhead, and how co-locating compute with vector storage enables fast retrieval with filtering and enrichment in a single runtime. The goal is a unified, stateful platform that reduces architectural sprawl while improving latency and resilience.

Key Discussion Points:

• Why most enterprise RAG systems are not truly real-time
• Trade-offs between batch, micro-batch, and streaming AI architectures
• How streaming pipelines enable continuous ingestion and embedding
• Architectural patterns for distributed ML inferencing at scale
• How in-memory vector collections support resilient low-latency semantic search

​We use the Luma site to host our events. SIGN UP HERE