Note
Key concepts
Before diving into the combine
command, it may be helpful to brush up on the concept of
configuration layers.
ksconf combine¶
Merge .conf settings from multiple source directories into a combined target
directory. Configuration files can be stored in a /etc/*.d
like directory
structure and consolidated back into a single ‘default’ directory.
This command supports both one-time operations and recurring merge jobs. For example, this command can be used to combine all users’ knowledge objects (stored in ‘etc/users’) after a server migration, or to merge a single user’s settings after their account has been renamed. Recurring operations assume some type of external scheduler is being used. A best-effort is made to only write to target files as needed.
The ‘combine’ command takes your logical layers of configs (upstream, corporate,
Splunk admin fixes, and power user knowledge objects, …) expressed as
individual folders and merges them all back into the single default
folder
that Splunk reads from. One way to keep the ‘default’ folder up-to-date is
using client-side git hooks.
No directory layout is mandatory, but taking advantages of the native-support
for ‘dir.d’ layout works well for many uses cases. This idea is borrowed from
the Unix System V concept where many services natively read their config files
from /etc/*.d
directories.
Version notes: dir.d was added in ksconf 0.8. Starting in 1.0 the default will switch to ‘dir.d’, so if you need the old behavior be sure to update your scripts.
usage: ksconf combine [-h] [--target TARGET] [-m {auto,dir.d,disable}] [-q]
[-I PATTERN] [-E PATTERN] [--dry-run] [--follow-symlink]
[--banner BANNER] [-K KEEP_EXISTING] [--disable-marker]
[--disable-cleanup]
source [source ...]
Positional Arguments¶
source | The source directory where configuration files will be merged from.
When multiple source directories are provided, start with the most general and end
with the specific; later sources will override values from the earlier ones.
Supports wildcards so a typical Unix conf.d/##-NAME directory structure works well. |
Named Arguments¶
--target, -t | Directory where the merged files will be stored. Typically either ‘default’ or ‘local’ |
-m, --layer-method | |
Possible choices: auto, dir.d, disable
| |
-q, --quiet | Make output a bit less noisy. This may change in the future… |
-I, --include | Name or pattern of layers to include. |
-E, --exclude | Name or pattern of layers to exclude from the target. |
--dry-run, -D | Enable dry-run mode. Instead of writing to TARGET, preview changes as a ‘diff’. If TARGET doesn’t exist, then show the merged file. |
--follow-symlink, -l | |
Follow symbolic links pointing to directories. Symlinks to files are always followed. | |
--banner, -b | A banner or warning comment added to the top of the TARGET file. Used to discourage Splunk admins from editing an auto-generated file. For other on-going combine operations, it’s helpful to inform any .conf file readers or potential editors that the file is automatically generated and therefore could be overwritten again.
For one-time combine operations, the default banner can be suppressed by passing in an empty string ( |
-K, --keep-existing | |
Existing file(s) to preserve in the TARGET folder. This argument may be used multiple times. | |
--disable-marker | |
Prevents the creation of or checking for the .ksconf_controlled marker file safety check.
This file is typically used indicate that the destination folder is managed by ksconf.
This option should be reserved for well-controlled batch processing scenarios. | |
--disable-cleanup | |
Disable all file removal operations. Skip the cleanup phase that typically removes files in TARGET that no longer exist in SOURCE |
You may have noticed similarities between the combine
and merge
subcommands. That’s because under the covers they are using much of the same code. The combine
operation essentially does a recursive merge between a set of directories. One big difference is
that combine
command will handle non-conf files intelligently, not just conf files.
Additionally, combined
can automatically detect layers for you, depending on the layering scheme in use.
Mixing layers¶
Just like all layers can be managed independently, they can also be combined in any way you would like.
This also allows for different layers to be mixed-and-matched by selectively including layers to combine.
This feature is now available in ksconf 0.8.0 and later using the --include
and --exclude
CLI options,
which should behave as just as you’d expected.
Note
A more detailed explanation
The --include
and --exclude
arguments are processed in the order given.
These filters are applied to all layer names.
The last match wins.
If --include
is first, then by default all layers, except for the ones explicitly included, will be excluded.
Conversely, if --exclude
is first, then all layers will be included except for the ones explicitly included.
If no filters are given then all layers will be processed.
Here’s an example, truncated for brevity, to further demonstrate how this can be used practically:
Splunk_TA_nix/
├── README.txt
├── bin
│ ├── bandwidth.sh
│ ├── common.sh
├── default.d
│ ├── 10-upstream
│ │ ├── app.conf
│ │ ├── data
│ │ │ └── ui
│ │ │ ├── nav
│ │ │ │ └── default.xml
│ │ │ └── views
│ │ │ └── setup.xml
│ │ ├── eventtypes.conf
│ │ ├── inputs.conf
│ │ ├── props.conf
│ │ ├── tags.conf
│ │ ├── transforms.conf
│ │ └── web.conf
│ ├── 20-common
│ │ ├── inputs.conf
│ │ ├── props.conf
│ │ └── transforms.conf
│ ├── 30-master-apps
│ │ └── inputs.conf
│ └── 30-shcluster-apps
│ ├── inputs.conf
│ └── web.conf
├── lookups
│ ├── nix_da_update_status.csv
│ ├── nix_da_version_ranges.csv
└── metadata
└── default.meta
Here we have several named layers in play:
10-upstream
- the layer used to contain the default app content that ships from the Splunk TA, or whatever is “upstream” source is.20-common
- organizational level change to deployed everywhere.30-master-apps
- The bits that should just go to the indexers.30-shcluster-apps
- Content that should go to just the search heads.
In this case, we always want to combine the 10-*
and 20-*
layers, but only want to include either the master or searchhead cluster layer depending on server role.
ksconf combine src/Splunk_TA_nix --target build/shcd/Splunk_TA_nix \
--exclude=30-* --include=30-shcluster-apps
ksconf combine src/Splunk_TA_nix --target build/cm/Splunk_TA_nix \
--exclude=30-* --include=30-master-apps
# Say you just want the original app, for some reason:
ksconf combine src/Splunk_TA_nix --target /build/orig/Splunk_TA_nix --include=10-upstream
Using this technique you can pretty quickly write some simple shell scripts to build these all at once:
for role in shcluster master
do
ksconf combine src/Splunk_TA_nix \
--target build/${role}/Splunk_TA_nix \
--exclude=30-* --include=30-${role}-apps
done
Hopefully this gives you some ideas on how you can start to build some custom workflows with just a few small shell scripts.
Layer methods¶
Ksconf supports different methods of layer detection mechanism. Right now just two different schemes are supported, but if you have other ways of organizing your layers, please reach out.
- Directory.d (
dir.d
)Also known as
*.d
directory layout is allows layers to be embedded on a directory structure that allows for simple prioritization and labels to be applied to each layer. Anyone who’s configured a Linux server should find this familiar.Example:
MyApp/default.d/10-my_layer/props.conf
Convention:
<directory-name>.d/<##>-<layer-name>/
When these layers are combined, the top level folder is modified to remove the trailing
.d
, and all content from the enable layers is combined within that folder. The layer-name portion of the path is discarded in the final combined path. Content is combined based on the assigned ranking of each layer, or directory sort order.- Disable (legacy)
- If you would prefer to stick with the previous behavior (no automatic detection of layers) and specify all SOURCE directories manually, then use this mode. In this mode, each layer must be explicitly defined (or provide as a wildcard) and any other files operations must be handled elsewhere.
- Auto (default)
- In auto mode, if more than one source directory is given, then
disable
mode is used, if only a single directory is given thendir.d
will be used.
How do I pick?¶
Mode | Useful when | Avoid if |
---|---|---|
dir.d |
|
|
disable |
|
|
Examples¶
Merging a multilayer app¶
Let’s assume you have a directory structure that looks like the following. This example features the Cisco Security Suite.
Splunk_CiscoSecuritySuite/
├── README
├── default.d
│ ├── 10-upstream
│ │ ├── app.conf
│ │ ├── data
│ │ │ └── ui
│ │ │ ├── nav
│ │ │ │ └── default.xml
│ │ │ └── views
│ │ │ ├── authentication_metrics.xml
│ │ │ ├── cisco_security_overview.xml
│ │ │ ├── getting_started.xml
│ │ │ ├── search_ip_profile.xml
│ │ │ ├── upgrading.xml
│ │ │ └── user_tracking.xml
│ │ ├── eventtypes.conf
│ │ ├── macros.conf
│ │ ├── savedsearches.conf
│ │ └── transforms.conf
│ ├── 20-my-org
│ │ └── savedsearches.conf
│ ├── 50-splunk-admin
│ │ ├── indexes.conf
│ │ ├── macros.conf
│ │ └── transforms.conf
│ └── 70-firewall-admins
│ ├── data
│ │ └── ui
│ │ └── views
│ │ ├── attacks_noc_bigscreen.xml
│ │ ├── device_health.xml
│ │ └── user_tracking.xml
│ └── eventtypes.conf
├── lookups
├── metadata
└─── static
In this structure, you can see several layers of configurations at play:
- The
10-upstream
layer appears to be the version of the default folder that shipped with the Cisco app.- The
20-my-org
layer is small and only contains tweaks to a few saved search entries.- The
50-splunk-admin
layer represents local settings changes to specify index configurations, and to augment the macros and transformations that ship with the default app.- And finally,
70-firewall-admins
contains some additional view (2 new, and 1 existing). Note that sinceuser_tracking.xml
is not a.conf
file it will fully replace the upstream default version (that is, the file in10-upstream
)
You can merge all these layers inside this app into a new app folder using the command below:
ksconf combine repo/Splunk_CiscoSecuritySuite --target=shcluster/apps/Splunk_CiscoSecuritySuite
ksconf
will automatically detect the default.d
folder as a layer-containing directory and merge content from the detected layers (10-upstream
, 20-my-org
, …) into a new default
folder in the resulting app.
All other content (such as README, bin, static, lookups and so on) will be copied as-is.
Changed in version 0.8: If you are using ksconf
before 0.8, then you have to manually merge the layers, and possibly copy other top-level folders on your own (outside of ksconf).
The example below still works fine after version 0.8, but the default behavior may change in 1.0, so it’s advisable to start using --layer-method
explicitly in any scripts you may use.
Here are the commands that could be used to generate a new (merged) default
folder from all
of the layers shown above.
cd Splunk_CiscoSecuritySuite
ksconf combine default.d/* --target=default
Note that in the example above, the default
folder now lives along side the default.d
folder.
Also note that only the contents of default.d
are copied, not the entire app, like in the above example.
See also
The unarchive command can be used to install or upgrade apps stored in a version controlled system in a layer-aware manor.
Consolidating ‘users’ directories¶
The combine
command can consolidate ‘users’ directory across several instances after a phased server migration.
See Migrating the ‘users’ folder.