Troubleshooting ELK Elasticsearch & Logstash Pt 1 of 2

How to Troubleshoot elasticsearch

You might find yourself attempting to view Elasticsearch logs through Kibana and realize that you have not been receiving logs for quite some time. Unfortunately, elasticsaerch might run into issues and will stop working if the following occur:

  • High load from queries (turning indexes red)
  • Low disk space (running out of space)

How to check if elasticsearch indexes are in a “red” state.

Method 1 – Terminal 

Run the following on your Linux terminal

curl elasticsearchserver:9200/_cat/indices?v

you should see “red” under health.

Alternatively If you have a cluster, you can check the cluster health:

curl 'http://elasticsearchserver:9200/_cluster/health'

you should see your Cluster Name, along with a status”:Red”.

Deleting “Red” Index. 

If you have a red index, you can manually remove it by doing the following:

Example: Curl –XDELETE http://elasticsearchserver:9200/indexname

curl -XDELETE http://elasticsearchserver:9200/logstash-winlogbeat-2017.08.07

Re-run the index query and ensure there are no other red indexes.

Method2 – Kibana 

Login to your Kibana instance http://elasticsearchserver:5601 and navigate to “Dev Tools”

and paste the following:

GET /_cat/indices?v&s=docs.count:desc

Click the  the Green play button 

You should see the following:

If you do find a red index, you may delete it from here.

Type the following in the console: delete yourindexname

You should see an “acknowledged: True” message to verify the deletion.

 

Last Resort: Ran out of disk space

If you ran out of disk space, elasticsearch might just not work at all. You may try to do a curl command, but you might receive a message stating that elasticsearch is not accessible via terminal, or even Kibana.

To verify the disk space in your node run the following command:

df –h

 

 

 

 

If your elasticsearch node is at 100% capacity, you may have to delete some files manually.

Elasticsearch files will be saved in the following location: /var/lib/elasticsearch/nodes/0/indices

Run the following:

cd /var/lib/elasticsearch/nodes/0/indices

You may run the following to see which folders have the most data

du –h

Once you find a folder with x amount of space, you may delete it to clear some space. (Note this might delete unexpected records from a certain time-period, so be careful)

Delete the folder by performing the following:

rm –rf WZbVdoWhSw2GH9ubzoiLSQ/

WzbvdoWhSw2.. is my folder name, change it to yours.

Run df –h to verify that the file was deleted and your disk capacity increased.

Lastly, restart the elasticsearch service:

Service elasticsearch restart

Next Article will go over troubleshooting Logstash.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments