Incorporating Virustotal Data to Elasticsearch

Now that we’re collecting logs from various sources including Sysmon, we have access to file hash information. A while back I came across this SANS article on incorporating Virustotal to Elasticsarch here. The article is from 2015 when Elasticsearch was still in version 1.5, the creator was Jason Kendall and the github project is posted here. Unfortunately I have tried to get this working on Logstash version 5.x without success; Therefore, I stayed with version 2.3.4. My test environment was running Elasticsaerch version 2.3.4 as well as Logstash 2.3.4 in which the Virustotal plug-in did work!

Since my environment is on Elasticsearch version 5.x, I ended up creating a small VM with Logstash 2.3.4 and setup the virustotal plugin here, the data is still going to my current server so I’m happy with that.

Prerequisites:

• Logstash 2.3.4 or earlier
• Git  (yum install git)
• Ruby Gem
• Nano (my preferred text editor)
• Sign up for a Virustotal public API https://www.virustotal.com

Step 1: Install Ruby On Rails

sudo yum install ruby

Say Yes to the promt

sudo yum install gcc g++ make automake autoconf curl-devel openssl-devel zlib-devel httpd-devel apr-devel apr-util-devel sqlite-devel
sudo yum install ruby-rdoc ruby-devel

Install Ruby Gems & Update

sudo yum install rubygems

gem update –system

Step 2: Install Virustotal Plug-in

cd /tmp
git clone https://github.com/coolacid/logstash-filter-virustotal.git
cd logstash-filter-virustotal

Since this script was written for versions prior to 2.0, we’re going to edit the .gemspec file.

 nano /tmp/logstash-filter-virustotal/logstash-filter-virustotal.gemspec

Edit the following, and change it to <3.0.0 or whatever version you choose.
s.add_runtime_dependency ‘logstash-core‘, ‘>= 1.4.0‘, ‘< 2.0.0‘

Save it (CTRL-O) and exit.
let’s continue with the installation.

gem build logstash-filter-virustotal.gemspec 
cd /opt/logstash
bin/plugin install /tmp/logstash-filter-virustotal/logstash-filter-virustotal-0.1.1.gem

Note: you might get a warning message of “fatal: Not a git repository (or any of the parent directories): .git” which you may ignore.

You may now restart logstash.

Lastly, confirm that your Virustotal plugin is part of the logstash plugins.
under /opt/logstash run the following:

bin/plugin list

You should see logstash-filter-virustotal listed

Done!

Now let’s configure Logstash to lookup file hashes that are being generated from Sysmon.

In my use case I’m only looking focusing on EventID 15 in Sysmon
Event ID 15: FileCreateStreamHash

My Logstash configuration does the following:

• Only looks at Event ID 15 and Hash field.
• Focuses on the highlighted information from Virustotal. (Note: when the file queries Virustotal, the Anti-virus list will also be populated; therefore, my logstash configuration will automatically remove that information).

Here’s my logstash configuration for Virustotal. (Note: I”m using NXlog to send the logs from my Windows system to Logstash.

#Winlogs Input
input {
  tcp {
    port => 5044
    type => "winlogs"
    codec => json
  }
}

#Sysmon Logs

# Sysmon remove Sha256= from field FileStream

filter {
if [type] == "winlogs" {
  if [Channel] == "Microsoft-Windows-Sysmon/Operational" and [EventID] == 15 {
  mutate {
    gsub => ["Hash", "SHA256=", ""]
  }
  }
}
}


# Sysmon virustotal check

filter {
if [type] == "winlogs" {
  if [Channel] == "Microsoft-Windows-Sysmon/Operational" and [EventID] == 15 
  and ([TargetFilename] =~ /(?i)\.(doc|zip|exe)/ ) {
  virustotal {
            apikey => 'MyApiKeyFromVirustotal'
            field => '[Hash]'
            lookup_type => 'hash'
            target => 'virustotal'
  }
  }
}
}

#Removes Virustotal AV Vendor Info
filter {
if [type] == "winlogs" {
  if [Channel] == "Microsoft-Windows-Sysmon/Operational" and [EventID] == 15 {
   mutate { remove_field => ["[virustotal][scans]"]}
  }
}
}

output {
if [type] == "winlogs" {
  elasticsearch { 
     hosts => ["http://MyElasticsearch5servers:9200"]
     index => "logstash-virustotal-%{+YYYY.MM.dd}"
 }
}
}

Update: 07-06-2017 – Logstash configuration to receive from Winlogbeat

input {
  beats {
    port => ##
	tags => [ "winlogbeat" ]
  }
}


#Drops non-hash files
filter {
if "winlogbeat" in [tags] and [log_name] == "Microsoft-Windows-Sysmon/Operational" and [event_id] == 15
  and [event_data][Hash] == "Unknown" {
    drop { }
  }
}

# Sysmon remove Sha256= from field FileStream

filter {
if "winlogbeat" in [tags] and [log_name] == "Microsoft-Windows-Sysmon/Operational" and [event_id] == 15 {
  mutate {
    gsub => ["[event_data][Hash]", "SHA256=", ""]
  }
  }

}

filter {
if "winlogbeat" in [tags] and [log_name] == "Microsoft-Windows-Sysmon/Operational" and [event_id] == 15 and ([event_data][TargetFilename] =~ /(?i)\.(doc|zip|exe)/ ) {
 virustotal {
           apikey => 'YourVirusTotalAPIKey'
           field => '[event_data][Hash]'
           lookup_type => 'hash'
           target => 'virustotal'
 }
 }
}


output {
if "winlogbeat" in [tags] and [log_name] == "Microsoft-Windows-Sysmon/Operational" and [event_id] == 15 {
   gelf {
      host => ["Your2.xLogstashserver"]
	  port => ##
   }
}
}

output {
if "winlogbeat" in [tags] and [log_name] == "Microsoft-Windows-Sysmon/Operational" and [event_id] == 15  {
  elasticsearch { 
     hosts => ["http://YourElasticsearchnode:9200"]
     index => "logstash-virustotal%{+YYYY.MM.dd}"
 }
}
}

Finally, here’s a sample log of Putty.exe without the unnecessary anti-virus vendor information.

Here’s a sample dashboard of Elasticsearch, which makes it easy to sort by high number of positives.

One last thing. Virustotal limits you to  the following if you have a public key:

Therefore please ensure that you are not abusing the daily limit; otherwise, you might need to upgrade to a Private API.

Here’s a sample of my current API Consumption:

Thanks for reading.