Splunk SNMP data
Splunk SNMP data was the first use case I thought of when I met Splunk.
Oh the grandiose plans I had for killing my various Cacti installs with this new, fancy, dynamic tool. All the customization I wanted was already done for me in Splunk. Finally, I could just focus on working with the data.
The potential was limitless.
That was 3 years ago.
In that time I spoke to many a Splunk enthusiast and one thing was abundantly clear. There was no “standard” way to Splunk SNMP. It was that one thing that everyone did differently. Different pollers, different environments and custom solutions. There was no simple one line answer.
And my Cacti installs lived on….
New Year, New SNMP
This year, my new year’s resolution was to finally build THE way for me to reliably and efficiently collect SNMP data en mass (100K sources +) and ingest into Splunk.
The one thing I, and all the Splunkers I spoke with agreed on was, the polling of SNMP data was best left to a downstream agent. Just feed Splunk the output and let it do what it does best.
I set out and tested various SNMP pollers – Cacti, rrdtool itself, collectd [ shout out to collectd for nailing the flexible output options, but the non-cron poller – opened a ticket/feature request on it – and the config management…I just couldn’t go that route in my environment], OpenNMS [ sweet option for writing to an interface, but again was worried at scale ], Solarwinds [too much for what I really needed], Splunk SNMP modular Input [ Again, solid job, but no cron made in non-viable for me ], etc, etc, etc.
I was looking for the perfect mix of easy SNMP administration/supportability and reliable output formatting for Splunk. Plus, I already had multiple reliable Cacti deployments with 4+ years uptime, with over 50k data sources each, so if there was a way I could retrofit something to use that existing infrastructure it would be a big win.
In my opinion and to my surprise, old faithful Cacti, who I had set out to destroy, still reigned supreme when it came to easy, reliable, management of massive amounts of SNMP, but for one major flaw….
SNMP polling output.
How would we get the data out of Cacti and into Splunk??
Mirage is born
I enlisted the help of my some amazing friends and colleagues and after a failed attempt at ingesting rrd files, soon it had dawned on us:
A CACTI PLUGIN!!
How I didn’t think of this 3 years ago is beyond me.
We utilized Cacti’s API architecture to design a plug-in that mirrors Spine SNMP poller data to log file before it is written to rrds, outputting beautifully structured, easy to consume KV pairs.
Mirage was born.
We then built Splunk add-on to help enrich the data we get, leveraging Cacti’s treasure trove of data sitting in it’s MySQL db.
A week later, when we stood back and watched it all at work, both in a large ISP production environment and at home in our mini-labs, it was clear we could finally cross Splunk SNMP off our whiteboards.
Over the next couple months, we will be hard at work iterating over the monster we have created, and we hope we can get some feedback from, what I believe to be, a large community of sysadmins looking for a way to leverage all that work they did building out Cacti, as we move to the next generation of IT Ops monitoring and intelligence tool-sets.
Splunk SNMP data with Cacti Mirage and let us know what is working and what opportunities there are to make Mirage even better!
ENOUGH READING, JUST TRY IT!!
To Splunk SNMP data with Cacti Mirage as your acquisition layer, you will require:
- Cacti & Splunk, installed and running
- If you need info on how to install Cacti, Cacti Spine or Splunk, here are some guides that should help:
- You need Cli with Sudo/root privileges & Admin access for Cacti/Splunk
- Cacti Mirage Plug-in [ Github or Cacti.org ]
- Cacti Mirage Add-on for Splunk
INSTALL CACTI’S NEW MIRAGE PLUG-IN
Before you can Splunk SNMP data, you must acquire it by installing the Mirage Plug-in for Cacti. Mirage allows you to mirror Cacti poller data to log file for easy ingestion into other systems. This also allows you to breathe new life into your Cacti installs by using Cacti as your more than capable SNMP poller (assuming you are using spine), while letting more modern upstream systems handle the rest.
Connect to the Cacti server cli and navigate to CACTI_HOME/plugins ( likely /usr/share or /var/www )
wget -O mirage.zip https://codeload.github.com/n00badmin/mirage/zip/master
mv mirage-master mirage
chown -R www-data:www-data mirage
Log into the Cacti GUI and navigate to the Console > Plugin Management.
Click to install mirage, then to enable it.
Navigate to Console > Settings
Review the Mirage settings and save.
Wait until next Cacti poll and ensure the mirage_poller_output.log is populated in CACTI_HOME/cacti/log/
root@n00bserver:/var/www/cacti/log# ls -la
drwxr-xr-x 2 www-data www-data 4096 Jan 29 00:45 .
drwxr-xr-x 14 www-data www-data 4096 Jan 26 10:19 ..
-rw-r–r– 1 www-data www-data 93118 Jan 29 01:45 cacti.log
-rw-r–r– 1 www-data www-data 33 Jul 19 2015 .htaccess
-rw-rw-r– 1 www-data www-data 68326 Jan 29 01:45 mirage_poller_output.log
SPLUNK SNMP – INSTALL CACTI MIRAGE ADD-ON FOR SPLUNK
Now to install the Splunk Cacti Mirage TA. The following will describe a stand-alone Splunk enterprise deployment. For distributed environments, install Cacti Mirage TA on SH, FW & IX.
Start by creating a ‘cacti’ index on your indexer.
Settings > index > New
Name it ‘cacti’ and save with all defaults.
Return to the CLI
wget -O Splunk_TA_Cacti.zip https://codeload.github.com/mennova/Splunk_TA_Cacti/zip/master
cp inputs.conf /opt/splunk/etc/apps/Splunk_TA_Cacti-master/local
Check to ensure the paths to the mirage_poller_output.log, cacti_lookup_mirage.py & cacti.log are correct for your environment, and ensure to set them to disabled= false. Set the cacti_lookup_mirage.py cron to fire as desired (example below fires every day at 6 am). This scripted input must run to ensure the Splunk lookups can enrich your SNMP data, so schedule it soon after the time you finish editing the inputs if you want to play right away!
disabled = false
index = cacti
sourcetype = cacti:mirage
disabled = false
index = cacti
sourcetype = cacti:system
source = cacti_lookup_mirage.py
disabled = false
index = cacti
sourcetype = cacti:lookup:mirage
interval = * 6 * * *
#interval = 86400
Save the inputs.conf
Once Splunk restarts, and your configured cron has fired, log into the Cacti Gui and confirm cacti_lookup_mirage.py has run by examining the ‘Cacti Polling & Lookups Status‘ dashboard. If you do not see it, go back and alter the cron in cacti_lookup_mirage.py and restart, or wait till it’s next run.
Now you can run the ‘Cacti lookup mirage build‘ outputlookup to enrich your SNMP eventsOnce built you can now search the Cacti Quick chart or use SPL to chart your data.
For a quick browse of what you have ingested try:
eventtype=cacti:mirage host=n00bserver | top limit=0 name_cache
- ldi (local_data_id) – This is the unique data souce id in Cacti
- rrdn (rrd_name) – The type of KPI/measurement in the rrd
- rrdv (rrd_value) – The value the poller was writing to the rrd.
- name_cache – Cacti Data Source Name/Name of the graph
- data_source_type_id – 1 = Gauge, 2= Counter
- host – The Cacti Server
- hostname – The device Cacti is querying
- IP – IP address of the hostname
For those who may want to play with the local_data_id lookup/db mappings:
Sample Splunk search for graphing cpu on a server
eventtype=cacti:mirage rrdn=cpu hostname=n00bserver
| timechart max(rrdv) by name_cache
You now have the ability to utilize Cacti collected SNMP data in Splunk!
Stay tuned for how to monitor this data with Splunk Machine Learning!