Cloud Austin Meetup

Thank you to our sponsors!

• Q2 is providing the venue!
• The organizers are providing drinks.

Kunal Sinha (45m) Lessons learned: replacing elastic search with a highly scalable alternative while using event-driven Architecture.

Kunal was recently involved in an effort to build a new platform for serving ultra-low latency, high-throughput data retrieval API requests. As part of this initiative, they moved away from Elastic Search due to its write-heavy indexing performance bottlenecks and designed a new event-driven architecture.

The new system leverages RedHat Debezium for change data capture, Kinesis for streaming, an in-house stream processor for real-time transformations, and Aurora PostgreSQL RDS for persistent storage.

In this talk, he'll detail the architecture of this highly scalable platform and share key learnings on managing eventual consistency in a distributed event-driven system, ensuring data integrity, and the operational challenges of building custom stream processing at scale.