Herding ELKs with consul.io

I recently had a need to put in place a scalable logging solution based around the ELK stack.

Issues (http://elasticsearch-users.115913.n3.nabble.com/Java-client-unable-to-connect-ClusterBlockException-td4019825.html) with (https://community.rackspace.com/products/f/18/t/4055) Multicast (http://jontai.me/blog/2013/03/elasticsearch-ec2-discovery/) networking (http://elasticsearch-users.115913.n3.nabble.com/discovery-multicast-vs-unicast-td3689223.html) aside, Elasticsearch scales pretty well on its own without the need for any additional overheads, however discovering whether a node is online or not and connecting only to available nodes can be tricky.

Scaling Logstash can be tricky, but it basically involves adding more Logstash servers (http://serverfault.com/questions/459303/scaling-logstash-with-redis-elasticsearch) to the mix and pointing them at your Elasticsearch cluster by defining multiple hosts (http://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-host) in your Logstash configuration.

Kibana (like most web applications) can only have one Elasticsearch host defined (http://www.elastic.co/guide/en/kibana/current/_setting_kibana_server_properties.html) in the config, so scaling out Kibana is more difficult.

The above raises the question – how do I know which Elasticsearch node to point my configuration at if I don’t know whether they are there or not.

The answer came in the form of consul.io (http://consul.io/). If you’ve not looked at Consul before then I can highly recommend it as a good way of providing rudimentary cluster status and service advertising with very little overhead.

In this talk we'll tell you more....

About our speaker

Matthew is a DevOps Engineer, spending most of his days on the command line helping customers deploy and maintain their code, whilst taking a “Test-Driven” approach to systems maintenance. In the past, Matthew has worked as a Python developer on the OpenStack project, managed a large-scale webhosting cluster and worked in a variety of other industries, all of which have shaped the way he approaches problems.