Logstash Master Script for ELK Health monitoring

The following is a master script that was created to check on the health for Logstash and Elasticsearch nodes. This is helpful if you don’t have x-pack setup in your environment.

The Script is ran using Cron (Scheduled job) every 2 minutes and will notify you if the following happened:

  1. If the IP Reputation files (maliciousIP.yaml) failed to download – Reference this article to create this file.
  2. If Elasticsearch nodes are in a “RED” state – Meaning that logs are not coming in
  3. If Logstash Disk Space is running low – Log parsing will most likely stop
  4. If Logstash ports are not listening (5050, 5065) – Logstash is not receiving logs.
  5. If Plug-in Error messages are generated in the Logstash logs

Note: I’m running Centos 7, and Logstash 5.x, and Python 2.7.5

Let’s go over the prerequisites first:

  1. Setup AlienVaultIPReputation.py which can be accomplished by following this article
  2. Setup email notification for the Logstash server
  3. Setup a Cron job to execute the master script.

1. Setup AlienVaultIPReputation

You can follow this article and get it setup. This script allows us to pull threat feeds from AlientVault and compare them real-time with our outbound IP addresses to see if there’s any callbacks to known malicious IP addresses in Elasticsearch.  This is an optional step, you may just comment out # the first portion of the master_script.sh script.

2. Setup email notification for the Logstash Server

We will first install mailx which is a fairly easy setup.

Step 1: Install mailx

Step 2: Edit configuration

set smtp=youremailserver.domain.com:25
set nss-config-dir=/etc/pki/nssdb/

Those are really the only setting you need.

Next, you may send a test email to ensure that this works.

Run the following command:

echo “Hey this is a test email” | mailx -s “Test email” pablo@domain.com

Note: You should get an authenticated email account from your exchange administrator and enter if it’s necessary (This is more secure); otherwise, you’ll need to add a relay on your exchange server.

3. Setup a Cron job to execute the master script.

Setting up Cron

Cron is a time-based job scheduler (Think scheduled task in windows), that allows you to scheduled any kind of bash script. We will be installing it and then adding our script to ensure that it runs in an hourly basis.

Step 1: Install Cron

Step 2: Configure

the */2 * * * *  means that this script will run every 2 minutes. If you need more info on Cron you can visit this page.

Additionally when this script runs it will write the results to a master_log.txt file so you can see the results when the script runs and ensure that there are no issues.

Master Script

Finally once you have setup the prerequisites, we can finally create the master script. Copy and save as master_script.sh and ensure that you edit your cron job so you are pointing to the right location. As you saw earlier, my master script is located under /home/pdelgado/master_script.sh

Final Notes

As I mentioned before this is very helpful if you aren’t running x-pack, or if you aren’t using other 3rd party tool to check the health status of your ELK nodes.

This is very simple to setup and it works great to notify if there’s an issue with the ELK setup.

Thanks to Arjun for contributing to the master script.