Splunk tips and tricks (Advanced)

Warning: This article is only for Splunk advance-users and admins!

As we discussed in our previous article (click here!) about basics of Splunk and its integration with Cisco FMC .

Simple searches look like the following examples. Note that there are literals with and without quoting and that there are data field as well as date source selections done with an "=":

Simple Filters

Two of the most important filters are "rex" and "regex". "rex" is for extraction of a pattern and storing it as a new field. This is why you need to specifiy a named extraction group in Perl like manner "(?…)" for example like this:

source="some.log" Fatal | rex "(?i) msg=(?P[^,]+)"

When you run the above query check the list of "interesting fields" it now should have an entry "FIELDNAME" listing you the top 10 fatal messages from "some.log" What is the difference to "regex" now? Well "regex" is like grep. Actually you can rephrase

source="pheniix_FW1.log" Fatal

to

source="pheniix_FW1.log" | regex _raw=".*Fatal.*"

and get the same result. The syntax of "regex" is simply "=". If you wanna filter for a specific field then it makes sense to use regex!

Capacity Planning Tool

There is an awesome online calculator at splunk-sizing.appspot.com. which is used frequently by careful Splunk designers!

Calculations

use this query in order to sum up a field for instance:

... | stats sum(<field>) as result | eval result=(result/1000)

Determine the size of log events by checking len() of _raw. The p30() and p50() functions are returning the 30 and 50 percentiles:

| eval raw_len=len(_raw) | stats avg(raw_len), p30(raw_len), p50(raw_len) by sourcetype

Some shortcuts

Splunk usually auto-detects access.log fields so you can do queries like:

source="/var/log/nginx/access.log" HTTP 500

source="/var/log/nginx/access.log" HTTP (200 or 30*)

source="/var/log/nginx/access.log" status=404 | sort - uri

source="/var/log/nginx/access.log" | head 1000 | top 50 clientip

source="/var/log/nginx/access.log" | head 1000 | top 50 referer

source="/var/log/nginx/access.log" | head 1000 | top 50 uri

source="/var/log/nginx/access.log" | head 1000 | top 50 method

Receive results per mail:

By appending "sendemail" to any query you get the result in your inbox!

... | sendemail to="office@pheniix.com"

Timecharts

Create a timechart from a single field that should be summed up

... | table _time, <field> | timechart span=1d sum(<field>)

... | table _time, <field>, name | timechart span=1d sum(<field>) by name

Inputs

splunk _internal call /data/inputs/tcp/raw

splunk _internal call /data/inputs/tcp/raw -get:search sourcetype=foo

splunk _internal call /servicesNS/nobody/search/data/inputs/tcp/raw/7092 -post:sourcetype bar -post:index bardata

Check licenses

use this query:

splunk list licenses

Index Statistics

List All Indices like this:

| eventcount summarize=false index=* | dedup index | fields index

| eventcount summarize=false report_size=true index=* | eval size_MB = round(size_bytes/1024/1024,2) |

REST /services/data/indexes | table title |

REST /services/data/indexes | table title splunk_server currentDBSizeMB frozenTimePeriodInSecs maxTime minTime totalEventCount

then on the command line you can call it:

$SPLUNK_HOME/bin/splunk list index

To query write amount of per index the metrics.log can be used:

index=_internal source=*metrics.log group=per_index_thruput series=* | eval MB = round(kb/1024,2) | timechart sum(MB) as MB by series

MB per day per indexer / index

index=_internal metrics kb series!=_* "group=per_host_thruput" monthsago=1 | eval indexed_mb = kb / 1024 | timechart fixedrange=t span=1d sum(indexed_mb) by series | rename sum(indexed_mb) as totalmb

index=_internal metrics kb series!=_* "group=per_index_thruput" monthsago=1 | eval indexed_mb = kb / 1024 | timechart fixedrange=t span=1d sum(indexed_mb) by series | rename sum(indexed_mb) as totalmb

How to reload apps?

Easy as a pie like this:

/debug/refresh

Debug Traces

You can enable traces per trace topic listed in splunkd.log. To change permanently edit /opt/splunk/etc/log.cfg and change the trace level from "INFO" to "DEBUG". Example:

category.TcpInputProc=DEBUG

You can also do it on the GUI in "System Settings"

Configuration

To list effective configuration

$SPLUNK_HOME/bin/splunk btool inputs list

To check configuration syntax

$SPLUNK_HOME/bin/splunk btool check

User Management

To reload authentication config from command line:

# At least for Splunk 6.x

splunk _internal call /authentication/providers/services/_reload -auth admin:changeme

# Older variant

splunk _internal rpc-auth ''

To list

splunk _internal call /services/authentication/roles -get:search indexes_edit

splunk _internal call /services/authentication/users -get:search john.smith

splunk _internal call /services/authentication/users/john.smith -method DELETE

Last but not least there is a great summary of Splunk search cheat sheets:

TRADEMARK LEGAL NOTICE

All product names, logos, and brands are the property of their respective owners in Austria or other countries. All company, product and service names used on this website are for identification purposes only. Pheniix is not affiliated with or an official partner of Cisco, CompTIA,Dimension Data, VMware, Amazon, Microsoft, Certified Ethical Hacker, (ISC)², Juniper, Wireshark, Offensive Security,Google, GNS3, F5, Python, Linux, Java, OpenStack, Vagrant, Ansible, Docker, GIT, , Blockchain or other companies. Use of these names, logos, and brands does not imply endorsement. The opinions expressed on pheniix are personal perspectives and not those of Cisco, Dimension Data or any other company. Pheniix runs as an independent blog.

#Splunk #SplunkEnterprise #SIEM #cisco #NationalCyberSecurityAwarenessMonth #Cybersecurity #Cybercrime #CCNACyberOps #CCNP #CCIE #CCIELab #MikeGhahremani