January Meetup

Happy new years!

We hope you spent the new years well and enjoyed christmas!

We've also got a treat for you: our long-time supporter and now co-organiser Jilles will give the first talk of the year! See the proposal below!

Jilles van Gurp - Using Elasticsearch as a database

Elasticsearch is a great search engine that indexes content and allows you to dig through it using a rich and sophisticated query engine. When I started using Elasticsearch in 2012 I came from a background of having built Lucene and Solr powered systems in various projects since 2003. However, to call me a search expert would be overstating the way I use this technology. And while I have of course learned a thing or two about using it and operating it, I'm by no means a data retrieval expert and instead consider myself more of a distributed systems plumber: I build scalable server stuff and Elasticsearch, Solr, and Lucene are means to an end to me. It happens to be a great at stuff I generally need in such systems and as such has become a near no brainer for me.

Elasticsearch is great for storing and searching stuff. These two functions can't be separated mostly. When you find stuff, you'll probably want to get your hands on the actual stuff as well. That's why Elasticsearch in its default configuration is a richly featured JSON document store with many features that are either hard or completely lacking in many other data stores. It needs to be able to do CRUD on all the things it searches through. Not only that but it needs to do a good job because running at scale means that re-indexing is actually a really disruptive and expensive thing to do. If it wasn't a reliable data store, it could not possibly be a reliable search engine. That means it is valid to ask how it stands up as a database in its own right.

In this presentation I give a hands on overview of what it means to use Elasticsearch as a database, what the risks and challenges are, and how you can mitigate those. Additionally, I will be presenting our newly OSSed java http client that we have been using for nearly three years to make life easier when doing so and talk you through several useful design patterns for using Elasticsearch, or similar data stores like e.g. Couch DB. I'll be talking about safe ways to update data, bulk indexing and updating strategies, efficiently migrating indices, managing index versions, and many other things.