Configuration layers

The idea of configuration layers is shared across multiple actions in Ksconf. Specifically, combine is used to merge multiple layers, and the unarchive command can be used to install or upgrade an app in a layer-aware way.

What’s the problem?

In a typical enterprise deployment of Splunk, a single app can easily have multiple logical sources of configuration:

  1. Upstream app releases, often from Splunkbase
  2. Organization-specific customizations or fixes added by a local developer
  3. Fixes to buggy upstream settings, like indexes.conf, requested by your Splunk admin
  4. Custom knowledge objects created by subject matter experts

Ideally we would like to version control these, but doing so is complicated because normally you have to manage all four of these logical layers in one ‘default’ folder.


Isn’t that what the local folder is for?

Splunk requires that app settings be located either in default or local; and managing local files with version control leads to merge conflicts. So effectively, all version controlled settings need to be in default, or risk merge conflicts. However, making changes to the default folder causes issues when you attempt to upgrade an app upstream. See how this is a dilemma?

Let’s suppose a new upstream version is released. If you aren’t managing layers independently, then you have to manually upgrade the app, being careful to preserve all custom configurations. Compare this to the solution provided by the combine functionality. The layered approach provides an advantage because logical sources can be stored separately in their own directories, thus allowing them to be modified independently. Using this approach, changes in the “upstream” layer will only come from an official release, and the organizational layer will contain customizations made solely by your organization. Practically, this means it’s no longer necessary to comb through commit logs identifying which custom changes need to be preserved and reapplied.

While this doesn’t completely remove the need for a person to review app upgrades, it does lower the overhead enough that updates can be pulled in more frequently, thus minimizing divergence.

Minimizing files

A typical scenario:

To customize a Splunk app or add-on, many admins simply copy the conf file from default to local and then apply changes to the local copy. That’s a common practice, but stopping there complicates future upgrades. The next step should be to clean up the local file, deleting all the unmodified entries that were copied from default.

Why does this matter?

If you’ve copied a default file into the local folder, this means that local file doesn’t contain only your settings, it contains a copy of all of the default settings too. So in the future, fixes published by the app creator are likely to be masked by your local settings. A better approach is to reduce the local conf file leaving only the stanzas and settings that you intended to change. While this is a monotonous to do by hand, it is easily accomplished by ksconf minimize. This makes your conf files easier to read and simplifies upgrades.

What does Splunk have to say about this? (From the docs)

“When you first create this new version of the file, start with an empty file and add only the attributes that you need to change. Do not start from a copy of the default directory. If you copy the entire default file to a location with higher precedence, any changes to the default values that occur through future Splunk Enterprise upgrades cannot take effect, because the values in the copied file will override the updated values in the default file.” – [SPLKDOC1]


It’s a good practice to minimize your files right away. If you wait, it may not be obvious what specific version of default that local was copied from. In other words, if you run the minimize command after you’ve upgraded the default folder, you may need to do extra work to manually reconcile upgrade differences, because any changes made between the initial version of the default file and the most recent release of the conf file cannot, be automatically addressed in this fashion. If your files are all in git, and you know a REF of the previous version of your default file, you can use some commands like this:

# Review the output of the log, and find the revision of the last change
git log --oneline -- default/inputs.conf

# Assuming "e633e6" was identified as the desired baseline ref, based on the 'log' output

# Compare what's changed in the 'inputs.conf' file between releases (FYI only)
ksconf diff <(git show e633e6:./default/inputs.conf) default/inputs.conf

# Now apply the 'minimization' based on the original version of inputs.conf
ksconf minimize --target=local/inputs.conf <(git show e633e6:./default/inputs.conf)

As always, be sure to double check the results.