ksconf filter

Filter the contents of a conf file in various ways. Stanzas can be included or excluded based on a provided filter or based on the presence or value of a key.

Where possible, this command supports GREP-like arguments to bring a familiar feel.

usage: ksconf filter [-h] [-o FILE] [--comments] [--verbose] [--skip-broken]
                     [--match {regex,wildcard,string}] [--ignore-case]
                     [--invert-match] [--files-with-matches]
                     [--count | --brief] [--stanza PATTERN]
                     [--attr-present ATTR] [--attr-matches ATTR PATTERN]
                     [--attr-not-matches ATTR PATTERN] [--empty-stanzas]
                     [-e | -d] [--keep-attrs WC-ATTR] [--reject-attrs WC-ATTR]
                     CONF [CONF ...]

Positional Arguments

CONF

Input conf file

Named Arguments

-o, --output

File where the filtered results are written. Defaults to standard out.

--comments, -C

Preserve comments. Comments are discarded by default.

--verbose

Enable additional output.

--skip-broken

Skip broken input files. Without this things like duplicate stanzas and invalid entries will cause processing to stop.

--match, -m

Possible choices: regex, wildcard, string

Specify pattern matching mode. Defaults to ‘wildcard’ allowing for * and ? matching. Use ‘regex’ for more power but watch out for shell escaping. Use ‘string’ to enable literal matching.

--ignore-case, -i

Ignore case when comparing or matching strings. By default matches are case-sensitive.

--invert-match, -v

Invert match results. This can be used to show what content does NOT match, or make a backup copy of excluded content.

Output mode

Select an alternate output mode. If any of the following options are used, the stanza output is not shown.

--files-with-matches, -l

List files that match the given search criteria

--count, -c

Count matching stanzas

--brief, -b

List name of matching stanzas

Stanza selection

Include or exclude entire stanzas using these filter options.

All filter options can be provided multiple times. If you have a long list of filters, they can be saved in a file and referenced using the special file:// prefix. One entry per line. Entries can be either a literal strings, wildcards, or regexes, depending on MATCH.

--stanza

Match any stanza who’s name matches the given pattern. PATTERN supports bulk patterns via the file:// prefix.

--attr-present

Match any stanza that includes the ATTR attribute. ATTR supports bulk attribute patterns via the file:// prefix.

--attr-matches, --attr-eq

Match any stanza containing ATTR == PATTERN. PATTERN supports the special file://filename syntax. Matching can be a direct string comparison (equals), or a regex and wildcard match.

Note that all --attr-match and --attr-not-match arguments are matched together. For a stanza to match, all rules must apply. If attr is missing from a stanza, the value becomes an empty string for matching purposes.

--attr-not-matches, --attr-ne

Match any stanza containing ATTR != PATTERN. See --attr-matches for additional details.

--empty-stanzas

Show only empty stanzas. This is incompatible with manyother attribute filter options.

-e, --enabled-only

Keep only enabled stanzas. Any stanza containing disabled = 1 will be removed. The value of disabled is assumed to be false by default.

-d, --disabled-only

Keep disabled stanzas only. The value of the disabled attribute is interpreted as a boolean.

Attribute selection

Include or exclude attributes passed through. By default, all attributes are preserved. Allowlist (keep) operations are preformed before blocklist (reject) operations.

--keep-attrs

Select which attribute(s) will be preserved. This space separated list of attributes indicates what to preserve. Supports wildcards.

--reject-attrs

Select which attribute(s) will be discarded. This space separated list of attributes indicates what to discard. Supports wildcards.

How is this different that btool?

Some of the things filter can do functionally overlaps with btool list. Take for example:

ksconf filter search/default/savedsearches.conf --stanza "Messages by minute last 3 hours"

Is essentially the same as:

splunk btool --app=search savedsearches list "Messages by minute last 3 hours"

The output is the same, assuming that you didn’t overwrite any part of that search in local. But if you take off the --app argument, you’ll quickly see that btool is merging all the layers together to show the final value of all attributes. That is certainly a helpful thing to do, but not always what you want.

Ksconf is only going to look at the file you explicitly pointed it to. It doesn’t traverse the tree on it’s own. This means that it works on app directory structure that live inside or outside of your Splunk instance. If you’ve ever tried to run btool check on an app that you haven’t installed yet, then you’ll understand the value of this.

In many other cases, the usage of both ksconf filter and btool differ significantly.

Note

What if I want a filter default & local at the same time?

In situations where it would be beneficial to filter based on the combined view of default and local, then simply use ksconf_cmd_merge first. Here are two options.

Option 1: Use a named temporary file

ksconf merge search/{default,local}/savedsearches.conf > savedsearches.conf
ksconf filter savedsearches.conf - --stanza "* last 3 hours"

Option 2: Chain both commands together

ksconf merge search/{default,local}/savedsearches.conf | ksconf filter --stanza "* last 3 hours"

Examples

Searching for attribute/values combinations

Find all enabled input stanzas with a sourcetype prefixed with apache:.

ksconf filter etc/apps/*/{default,local}/inputs.conf \
    --enabled-only --attr-eq sourcetype 'apache:*'

List the names of saved searches using potentially expensive search commands:

ksconf filter etc/apps/*/{default,local}/savedsearches.conf \
    -b --match regex \
    --attr-eq search '.*\|\s*(streamstats|transaction) .*'

Show sourcetype stanzas where EVENT_BREAKER is defined but not enabled:

ksconf filter etc/deployment-apps/*/{default,local}/props.conf \
    --skip-broken --match regex \
    --attr-match-equals EVENT_BREAKER '.+' \
    --attr-match-not-equals EVENT_BREAKER_ENABLE '(true|1)'

Note that both conditions listed must match for a stanza to match. Logical ‘AND’ not an ‘OR’. Also note the use of --skip-broken because sometimes Splunk base apps have invalid conf files.

Lift and shift

Copy all indexes defined within a specific app.

cd $SPLUNK_DB
for idx in $(ksconf filter $SPLUNK_HOME/etc/app/MyApp/default/indexes.conf --brief)
do
    echo "Copy index ${idx}"
    tar -czf "/migrate/export-${idx}" "${idx}"
done

Now you’ll have a copy all of the necessary indexes in the /migrate folder to make MyApp work on another Splunk instance. Of course, there’s likely other migration tasks to consider, like copying the actual app. This is just one way ksconf can help.

Can I do the same thing with standard unix tools?

Sure, go for it!

Yes, there’s significant overlap with the filter command and what you can do with grep, awk, or sed. Much of that is on purpose, and in fact some command line arguments were borrowed.

I used to do these tasks by hand, but it’s easy to make mistakes. The idea of ksconf is to give you stable and reliable tools that are more suitable for .conf file work. Also keep in mind that these features are expanding much more quickly than the unix tools change.

Although, if you’ve had to deal with BSD vs GNU tools and trying to find a set of common arguments, then you probably already appreciate how awesome a domain-specific-tool like this is.