In this series we will go ahead and setup Elasticsearch 5 to collect Windows Logs.
The point of this tutorial is to setup a test environment for Elasticsearch on a single node which will allow us to search through logs real-time and provide an ability to create meaningful visualizations from Kibana. We will be focusing on collecting Windows logs such as account lockouts, account login failures, task creation, firewall modification, process creations, network connections.
I would suggest to take 5 minutes to understand what The Elasticsearch Stack is by visiting their main site.
Our setup: 1 server (CentOS) setup with Static IP addresses.
———————————————————————————————————–
- 1 (server name: ELK) server will serve as our Logstash parser
Before moving forward please read the following pertaining to Memory Lock information as my guide will include both. If you’re not interested in the Memory Lock just skip the steps that pertain to it.
Here are the prerequisites for this tutorial:
- CentOS 7 minimal – Latest built (or full Dvd download)
- net-tools
- nano
- Java 1.8.x
First of all, download the latest CentOS 7 minimal to get started, and install it which is covered by this tutorial
Once you have installed the Operating System, you may then:
Run an OS update to ensure that you are getting the most up-to-date applications for YUM. (perform these steps for all servers)
1 |
sudo yum upgrade |
Ensure that you are installing basic network tools such as “ifconfig”
1 |
sudo yum install net-tools |
Run the following to get a text editor
1 |
sudo yum install nano |
Set Static IP addresses for ELK (Optional Step, you may just leave as DHCP)
1 2 |
cd /etc/sysconfig/network-scripts/ nano ifcfg-(Whatever your network interface is called) |
sample configuration:
1 2 3 4 5 6 |
TYPE="Ethernet" BOOTPROTO="static" IPADDR=192.x.x.x GATEWAY=192.x.x.x DNS1=192.x.x.x NETMASK=255.255.255.0 |
Reboot your network adapter.
1 |
systemctl restart network.service |
At this point I’d suggestion you create some DNS records for your systems.
Now that we’re done with basic CentOS 7 items, let’s move on to Elasticsearch Prerequisites.
Step 1: install Java 1.8 DJK.
Install on all servers
1 |
sudo yum install java-1.8.0-openjdk.x86_64 |
set $JAVA_HOME to
1 |
export JAVA_HOME=/usr/lib/jvm |
you may run “java -version” to confirm your version of java installed.
1 |
java -version |
Step 2: Import Elasticsearch PGP key
1 |
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch |
Step 3: Setup Repositories for Elasticsearch, Kibana, and Logstash
For Elasticsearch:
Navigate to /etc/yum.repos.d and create a new repository file, call it elasticsearch.repo
1 |
nano /etc/yum.repos.d/ |
And copy the following to it:
1 2 3 4 5 6 7 8 |
[elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md |
CTRL-O and save as elasticsearch.repo”
Do the same for Kibana
1 |
nano /etc/yum.repos.d/ |
1 2 3 4 5 6 7 8 |
[kibana-5.x] name=Kibana repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md |
Save as kibana.repo
Lastly, setup a repo for Logstash
1 |
nano /etc/yum.repos.d/ |
1 2 3 4 5 6 7 8 |
[logstash-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md |
save as logstash.repo
Step 4: install Elasticsearch:
1 |
sudo yum install elasticsearch |
Step 5: Set the service to start automatically
To configure Elasticsearch to start automatically when the system boots up, run the following commands:
1 2 |
sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable elasticsearch.service |
Elasticsearch can be started and stopped as follows:
sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service
Step 6: Setup Firewall Rules
Add firewall rules, (Kibana will run on port 5601, Elasticsearch will run on port 9200, 9300, and Logstash will be running on port 5044 or whichever port you decide)
server names: (ELK)
1 2 3 4 |
firewall-cmd --permanent --zone=public --add-port=5601/tcp firewall-cmd --permanent --zone=public --add-port=9200/tcp firewall-cmd --permanent --zone=public --add-port=9300/tcp firewall-cmd --permanent --zone=public --add-port=5044/tcp |
Step 7: Test Elasticsearch
run the following query:
curl -XGET ‘yourELKipaddress:9200/?pretty’
(note: this is your local ip address)
You should see the following:
{
“version” : {
“number” : “5.2.0”,
“build_hash” : “24e05b9”,
“build_date” : “2017-“,
“build_snapshot” : false,
“lucene_version” : “6.4.0”
},
“tagline” : “You Know, for Search”
}
Step 8: Configure Elasticsearch:
1 |
nano /etc/elasticsearch/elasticsearch.yml |
Edit the following options and ensure that you remove the #comment field to enable them.
1 2 3 4 5 6 |
cluster.name: yourclustername node.name: ELK path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 192.x.x.x http.port: 9200 |
The following crossed items are old and unnecessary steps (updated 09-25-2017).
Next, uncomment the following setting MAX_LOCKED_MEMORY=unlimited (Remove the #)
1 2 |
<del>nano /etc/sysconfig/elasticsearch MAX_LOCKED_MEMORY=unlimited</del> |
Save and exit
Again we’re doing it at the service level too, uncomment LIMITMEMLOCK=infinity
1 2 |
<del>nano /usr/lib/systemd/system/elasticsearch.service LIMITMEMLOCK=infinity</del> |
Save and exit
For this step, we’re going to edit the Xms and Xmx
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
Note: set the GB to half your System’s RAM (In this example, I have a total of 8GB RAM; therefore my settings will be the following:
1 2 3 |
nano /etc/elasticsearch/jvm.options -Xms4g -Xmx4g |
Save and Exit.
Run the following once you have edited both configurations:
1 2 |
systemctl daemon-reload systemctl restart elasticsearch |
Step 10: Install KIBANA
1 |
sudo yum install Kibana |
To configure Kibana to start automatically when the system boots up, run the following commands:
1 2 |
sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable kibana.service |
Kibana can be started and stopped as follows:
1 2 |
sudo systemctl start kibana.service sudo systemctl stop kibana.service |
Step 11: Configure Kibana
1 |
nano /etc/kibana/kibana.yml |
Change the following settings:
1 2 3 |
server.port: 5601 server.host: "ELKipaddress" elasticsearch.url: "http://ELKipaddress:9200" |
Save it, and then restart the Kibana service.
1 |
sudo systemctl restart kibana.service |
You should be able to visit: http://YourELK:5601
Step 12: Configure Logstash
Install Logstash
1 |
sudo yum install logstash |
Setup Logstash as a service
1 2 |
sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable logstash.service |
Next we will create a basic logstash configuration (assuming that you will be using Winlogbeats (Next tutorial) to send your data)
If you will be using Winlogbeat to send data use the following logstash configuration: (For this tutorial use the first configuration shown)
Here’s the location for The Logstash configuration: /etc/logstash/conf.d
This is where you would save the following configuration as filename.conf
1 2 3 4 5 6 7 8 9 10 11 12 13 |
input { beats { port => 5044 } } output { if [type] == "wineventlog" { elasticsearch { hosts => ["http://ELKipaddress:9200"] index => "logstash-winlogbeat-%{+YYYY.MM.dd}" } } } |
If you are sending logs using NXlogs you may use the following logstash configuration:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
input { tcp { port => 5044 type => winlogs codec => json } } output { elasticsearch { hosts => ["http://elastic2ipaddress:9200","http://elastic3ipaddress.44:9200","http://elastic1ipaddress:9200"] index => "logstash-winlogs" } } |
Save the configuration and restart the Logstash service.
1 |
service logstash restart |
Logstash will now listen on port 5044. You may run netstat -plnt to see the listening port:
you should see something similar to this:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::5044 :::* LISTEN 8595/java
Now you have completed the Elasticsearch (ELK) stack basic setup. On the next tutorial I will go over adding data from a windows system using Winlogbeat and creating Indexes from Kibana.
Leave a Reply
Be the First to Comment!