Cloud-native Search Engine for Log Management and Analytics

Abstract
“Cloud-native” is the approach to building and running applications to exploit the advantages of a cloud computing delivery model. For datastores, this means leveraging low-level cloud services (compute, storage, orchestration) to build systems that are more resilient, scalable, and elastic.

Traditional search engines, such as Elasticsearch, have proven limited in their ability to embrace the cloud paradigm because they adopt a shared-nothing architecture and use indexing technology that requires all data to live on local drives predominantly.

Quickwit is a new open-source search engine for log management and analytics, designed from the ground up to leverage the full power and promise of the cloud. Quickwit is the first search engine to decouple storage from compute, enabling users to run sub-second search queries over terabytes of data stored on object storage (Amazon S3, Azure Blob Storage, etc.).

This presentation will discuss the design decisions, architecture, and implementation strategies that make this possible.

Agenda
In this talk, Adrien will cover
- Cloud-nativeness limitations of traditional search engines
- Quickwit’s novel approach and architecture
- Production use cases of Quickwit
- Demo

Bio
Adrien is a co-founder and core contributor of Quickwit. He’s a software developer with more than a decade of experience working in small, medium, and large-scale startups in France and the U.S. Prior to founding Quickwit, Adrien was a software engineer at Airbnb on the data infrastructure team.