Indexing and Searching Logs with Elasticsearch or Solr

Hosted by Analytics.Club NYC Search & Discovery

Public group

This is a past event

127 people went

Location image of event venue


We’ll talk about using Elasticsearch or Solr to index your logs, so you can search and analyze them in real-time. The term “logs” can range from server logs and application events to metrics or even social media information.

The presentation will have three parts:

• Overview of logging tools that play nicely with Elasticsearch and Solr

• Performance tuning and scaling Elasticsearch and Solr

• Demo of an end-to-end solution

First, we’ll discuss how Logstash, Apache Flume or rsyslog can help you parse, buffer and ship your logs to Elasticsearch or Solr. We’ll show the pros and cons of each tool, so you can decide what’s best for your use case.

Then, we’ll put Elasticsearch and Solr side by side and show how to make them handle write-heavy applications like those presented in the first part. We’ll talk about how to tweak various buffers, merge policies, how to structure indices/collections, and more. And when one machine isn’t enough, we’ll look at how Elasticsearch and Solr (through SolrCloud) can be scaled horizontally to handle more and more data.

In the demo part, we’ll take one end-to-end solution from all the possible combinations discussed and start indexing, searching and analyzing a high flow of logs.