Rapid 7 Nexpose Data to Splunk

The following is a step-by-step approach to setup Nexpose Data Warehouse to export to a Windows PostGres Database setup and allow Splunk to import it. The current Rapid 7 Splunk App does not function 100% and all Nexpose customers using the Splunk App are missing vulnerability data.

A Rapid 7 App for Splunk has been available which relies on various python scripts and a Nexpose Api (2.x) which triggered to pull certain data. After testing this, I realized that it does not pull all data necessary such as the “Insight Agent” vulnerability data. When comparing reports from Nexpose to those on Splunk, I realized some of the asset data was missing.

See example:

Rapid 7 Splunk App: https://splunkbase.splunk.com/app/3457/

After further reading, I realized that at some point they will decommission their current API and will focus on their newer one. Now their 3.x API is intended for automation rather than performing massive data exports. Nexpose provides a “Data Warehouse” https://blog.rapid7.com/2016/11/24/dimensional-data-warehouse-export-part-of-nexpose-646/ which allows you to export all Asset/Vulnerability data to an external PostGres Database which provides you with more functionality.

This is the direction that I decided to go to and then leverage the Splunk DB-Connect to import the data into Splunk and have a complete set of information.

Below are the Steps to setup the PostGres Database and queries used to pull data into Splunk.

Prerequisites for a Windows Environment

 

  • Windows 2012 Server (Non-Server OS should also work).
  • PostGres Database Installer
  • Access to Splunk Host (Db-Connect App)
  • Admin access to the Nexpose Console
  • Note the IP address for Nexpose console and the Splunk Host where Db-Connect is setup

 

  1. Download and Install PostGres for Windows.

Download the latest version of PostGres for Windows.

https://www.postgresql.org/download/windows/

For this particular case we’re downloading the latest version (11).

Choose the installation path. In my case I created an extra partition specifically for the DB data.

D:\PostgreSQL\11\data

Set and record the password for the database superuser (Postgres).

You will get prompted to open “stack builder” you may click no.

Launch PGAdmin 4

Navigate to Servers > PostgresSQL 11

You will get prompted for the password you created earlier, go ahead and enter it.

Next: Create a nexpose login

Click on the “Definition” tab and type a password:

Username: nexpose
P: YourPassword

Under “privileges” go ahead and give it read rights

Save!

Now go back to the Databases and create a new database

 

Step 2:  Allow Inbound connections to Postgres Database on Windows Firewall

Next, we’re going to make an exception in the Windows firewall to ensure that our host is allowing incoming connections on port 5432.

2.1 Launch Windows Firewall with advanced Security

Right click on “Inbound Rules” and create New

Done.

Step 3: Edit the Postgres Client Authentication Configuration  (pg_hba.conf) file

The pg_hba.conf is the Client Authentication Configuration file which will allow us to trust connections coming from the Nexpose Console & the Splunk Host via Db-Connect. This step is crucial to ensure that we’re able to connect with our database.

Ensure that you allow connections from the Nexpose server so we may connect to the PostGres db.

Ensure that both Nexpose & Splunk Heavy Forwarder are allowed, especially if they reside in different subnets.

Host all all 192.168.1.0/24 trust

Step 4: Setup Nexpose console to export data to the PostGres DB Host

4.1 Navigate to Nexpose console via your favorite web browser.

Navigate to Administration

4.2 Look for “Maintenance, Storage and Troubleshooting” and click on “manage”

4.3 Enter the Server address and test the connection:

You should get a “Credentials were successful”

If you get other messages related to SSL, etc.. It probably means you didn’t enter the information correctly.

Next, setup a reoccurring schedule to ensure that new scan data is exported baed on your scanning schedule. In my example I set it to run every 12 hours.

Make sure you Save your settings at the top right

Now we wait.

Step 5: Build the SQL Queries

I wrote the following queries to match (or closely match) the fields returned and imported into splunk using the Rapid7-Splunk App. I’ve also included the Splunk Search index & sourcetype

1. Nexpose Software information

Scheduled export time: Cron: * 4 * * *

Splunk Search: index=nexpose sourcetype=”nexpose:software”

SQL Query:

2. Nexpose Vulnerability Information

Scheduled export time: Cron: 5 4 * * *

Splunk Search: index=nexpose sourcetype=”rapid7:nexpose:vuln”

 

SQL Query:


3. Nexpose Asset Information:

Scheduled export time: 10 4 * * *

Splunk Search: index=nexpose sourcetype=”rapid7:nexpose:asset”

Note: This is a very large query and should be set to 4:10 a.m. to ensure that the previous queries do not interfere.

Set timeout to: 600 seconds.

Note: This is a very large query and might take some time depending on the amount of data in the database.

SQL Query:

Step 6: Setup Splunk DB-Connect to pull data from PostGres Database.

 

  1. Create a Nexpose Identity
  • On Splunk Navigate to Identifies and create a new identity caleld “nexpose” and enter the password you created for the PostGres “Nexpose” user
  • Save it when done
  1. Create a Database Connection
  • Now on the Connections tab, click on “New Connection” and call it “nexpose”

  • Select the appropriate Identity and connection type set to “Postgressql”
  • Fill in the rest of the information that pertains to your environment

  • Save once you are done.
  1. Create a new input- Now that you’ve created an identity and connections in Splunk. Let’s create an input
  • Select your Connection & Database Name. In this case “public” schema. Although If you leave it at default it should also work.
  • Next paste your SQL query and “Execute SQL” to ensure that your query works.
  • Provide a meaningful name and set a cron job to schedule the queries
  • On the index information you may leave the following blank:
    – Host
    – Source
  • Fill in the information for
    Sourcetype: rapid7:nexpose:vuln
    Index: nexpose
  • Set your Fetch size to whatever makes sense to you
  • Done! Save and wait

Once the data imports to Splunk  you may now start querying and creating meaningful reports.

Leave a Reply

avatar
  Subscribe  
Notify of