ksconf package

Submodules

ksconf.archive module

class ksconf.archive.GenArchFile(path, mode, size, payload)

Bases: tuple

mode

Alias for field number 1

path

Alias for field number 0

payload

Alias for field number 3

size

Alias for field number 2

ksconf.archive.extract_archive(archive_name, extract_filter: callable = None) → Iterable[ksconf.archive.GenArchFile]
ksconf.archive.gaf_filter_name_like(pattern)
ksconf.archive.gen_arch_file_remapper(iterable: Iterable[ksconf.archive.GenArchFile], mapping: Sequence[Tuple[str, str]]) → Iterable[ksconf.archive.GenArchFile]
ksconf.archive.sanity_checker(iterable: Iterable[ksconf.archive.GenArchFile]) → Iterable[ksconf.archive.GenArchFile]

ksconf.cli module

KSCONF - Ksconf Splunk CONFig tool

Optionally supports argcomplete for commandline argument (tab) completion.

Install & register with:

pip install argcomplete activate-global-python-argcomplete (in ~/.bashrc)
ksconf.cli.build_cli_parser(do_formatter=False)
ksconf.cli.check_py()
ksconf.cli.check_py_sane()

Run a simple python environment sanity check. Here’s the scenario, if Splunk’s python is called but not all the correct environment variables have been set, then ksconf can fail in unclear ways.

ksconf.cli.cli(argv=None, _unittest=False)
ksconf.cli.handle_cmd_failed(subparser, ep)

Build a bogus subparser for a cmd that can’t be loaded, with the only purpose of providing a more consistent user experience.

ksconf.combine module

XXX: Move the overwrite-as-necessary logic into a subclass; for several use cases we just don’t care because ‘target’ is a brand new directory

class ksconf.combine.LayerCombiner(follow_symlink: bool = False, banner: str = '', dry_run: bool = False, quiet: bool = False)

Bases: object

Class to recursively combine layers (directories) into a single rendered output target directory. This is heavily used by the ksconf combine command as well as by the package command.

Typical class use case:

::

lc = LayerCombiner()

# Setup source, either
  1. lc.set_source_dirs() OR
  2. lc.set_layer_root()

Call hierarch:

lc.combine()                    Entry point
    -> prepare()                Directory, layer prep
        -> prepare_target_dir() Make dir; subclass handles marker here (combine CLI)
    -> pre_combine_inventory()  Hook for pre-processing (or alerting) the set of files to combine
    -> combine_files()          Main worker function
    -> post_combine()           Optional, cleanup leftover files
add_layer_filter(action, pattern)
combine(target)

Combine layers into target directory.

combine_files(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile])
conf_file_re = re.compile('([a-z_-]+\\.conf|(default|local)\\.meta)$')
filetype_handlers = [(<function LayerCombiner.register_regex.<locals>.match_f>, <function handle_merge_conf_files>), (<function LayerCombiner.register_regex.<locals>.match_f>, <function handle_spec_concatenate>)]
post_combine(target)

Hook point for post-processing after all copy/merge operations have been completed.

pre_combine_inventory(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile]) → List[ksconf.layer.LayerFile]

Hook point for pre-processing before any files are copied/merged

prepare(target: pathlib.Path)

Start the combine process. This includes directory checking, applying layer filtering, and marker file handling.

prepare_target_dir(target: pathlib.Path)

Hook to ensure destination directory is ready for use. This can be overridden to adder marker file handling for use cases that need it (e.g., the ‘combine’ command)

classmethod register_regex(regex_match)

Decorator that matches a filename regex and if it matches, it executes the decorator handler.

set_layer_root(root: ksconf.layer.LayerRootBase.Layer)
set_source_dirs(sources: List[pathlib.Path])
spec_file_re = re.compile('\\.conf\\.spec$')
exception ksconf.combine.LayerCombinerException

Bases: Exception

ksconf.combine.handle_merge_conf_files(context: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)
ksconf.combine.handle_spec_concatenate(context: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)

ksconf.compat module

Silly simple Python version compatibility items

ksconf.compat.cache(user_function)
ksconf.compat.handle_py3_kw_only_args(kw, *default_args)

Fake support for Python 3.8+ style keyword-only style arguments, or * arg syntax.

Example Python 3.8+ syntax:

def f(arg, *args, a=True, b=False):
    ...

Example Python 3.7 (and earlier) syntax with this helper function:

def f(arg, *args, **kw_only):
    a, b = handle_py3_kw_only_args(kw_only, ("a", True), ("b", False))
    ...
Parameters:
  • kw (dict) – keyword arguments provided to the calling function. Be aware that this dict will be empty after this function is done.
  • default_args (tuple) – pairs of keyword argument to the caller function in argument (arg_name, default_value) order.
Raises:

TypeError – if kw contains any keys not defined in args This mirrors Python’s native behavior when an unexpected argument is passed to a function.

ksconf.consts module

class ksconf.consts.SmartEnum

Bases: enum.Enum

An enumeration.

CREATE = 'created'
NOCHANGE = 'unchanged'
UPDATE = 'updated'

ksconf.filter module

class ksconf.filter.FilteredList(flags=0, default=True)

Bases: object

IGNORECASE = 1
INVERT = 2
VERBOSE = 4
feed(item, filter=None)
feedall(iterable, filter=None)
has_rules
match(item)
match_path(path)
match_stanza(stanza)

Same as match(), but handle GLOBAL_STANZA gracefully.

reset_counters()
class ksconf.filter.FilteredListRegex(flags=0, default=True)

Bases: ksconf.filter.FilteredList

Regular Expression support

calc_regex_flags()
reset_counters()
class ksconf.filter.FilteredListSplunkGlob(flags=0, default=True)

Bases: ksconf.filter.FilteredListRegex

Classic wildcard support (‘*’ and ?’) plus ‘…’ or ‘**’ for multiple-path components with some (non-advertised) pass-through regex behavior

class ksconf.filter.FilteredListString(flags=0, default=True)

Bases: ksconf.filter.FilteredList

Handle simple string comparisons

reset_counters()
class ksconf.filter.FilteredListWildcard(flags=0, default=True)

Bases: ksconf.filter.FilteredListRegex

Wildcard support (handling ‘*’ and ?’) Technically fnmatch also supports [] and [!] character ranges, but we don’t advertise that

ksconf.filter.create_filtered_list(match_mode, flags=0, default=True)

ksconf.layer module

class ksconf.layer.DirectLayerRoot(config: Optional[ksconf.layer.LayerConfig] = None)

Bases: ksconf.layer.LayerRootBase

A very simple direct LayerRoot implementation that relies on all layer paths to be explicitly given without any automatic detection mechanisms. You can think of this as the legacy implementation.

add_layer(path: pathlib.Path)
order_layers()
class ksconf.layer.DotDLayerRoot(config=None)

Bases: ksconf.layer.LayerRootBase

class Layer(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, config: ksconf.layer.LayerConfig, file_factory: Callable, prune_points: Optional[Set[pathlib.Path]] = None)

Bases: ksconf.layer.Layer

prune_points
walk() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]
apply_filter(layer_filter: ksconf.layer.LayerFilter)

Apply a destructive filter to all layers. layer_filter(layer) will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.

Returns True if layers were removed

layer_regex = re.compile('(?P<layer>\\d\\d-[\\w_.-]+)')
list_layers() → List[Layer]
mount_regex = re.compile('(?P<realname>[\\w_.-]+)\\.d$')
order_layers()
set_root(root: pathlib.Path, follow_symlinks=None)

Set a root path, and auto discover all ‘.d’ directories.

Note: We currently only support ‘.d/<layer>’ directories, a file like default.d/10-props.conf won’t be handled here.

class ksconf.layer.LayerConfig

Bases: object

exception ksconf.layer.LayerException

Bases: Exception

class ksconf.layer.LayerFile(layer: ksconf.layer.LayerRootBase.Layer, relative_path: pathlib.PurePath, stat: Optional[os.stat_result] = None)

Bases: os.PathLike

layer
logical_path
static match(path: pathlib.PurePath)
mtime
physical_path
relative_path
resource_path
size
stat
class ksconf.layer.LayerFilter

Bases: object

add_rule(action, pattern)
evaluate(layer: ksconf.layer.LayerRootBase.Layer) → bool
class ksconf.layer.LayerRootBase(config: Optional[ksconf.layer.LayerConfig] = None)

Bases: object

All ‘path’s here are relative to the ROOT.

class Layer(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, config: ksconf.layer.LayerConfig, file_factory: Callable)

Bases: object

Basic layer Container: Connects logical and physical paths.

config
get_file(path: pathlib.Path) → ksconf.layer.LayerFile

Return file object (by logical path), if it exists in this layer.

iter_files() → Iterator[ksconf.layer.LayerFile]
list_files() → List[ksconf.layer.LayerFile]
logical_path
name
physical_path
root
walk() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]
add_layer(layer: Layer, do_sort=True)
apply_filter(layer_filter: ksconf.layer.LayerFilter) → bool

Apply a destructive filter to all layers. layer_filter(layer) will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.

Returns True if layers were removed

get_file(path) → Iterator[ksconf.layer.LayerFile]

return all layers associated with the given relative path.

get_layers_by_name(name: str) → Iterator[ksconf.layer.LayerRootBase.Layer]
iter_all_files() → Iterator[ksconf.layer.LayerFile]

Iterator over all physical files.

list_files() → List[ksconf.layer.LayerFile]

Return a list of logical paths.

list_layer_names() → List[str]
list_layers() → List[Layer]
list_logical_files() → List[ksconf.layer.LayerFile]

Return a list of logical paths.

list_physical_files() → List[ksconf.layer.LayerFile]
order_layers()
exception ksconf.layer.LayerUsageException

Bases: ksconf.layer.LayerException

class ksconf.layer.TemplatedLayerFile(*args, **kwargs)

Bases: ksconf.layer.LayerFile

logical_path
static match(path: pathlib.PurePath)
physical_path
render(template_path: pathlib.Path) → str
resource_path
template_context = {}
static transform_name(path: pathlib.PurePath)
ksconf.layer.layer_file_factory(layer, path: pathlib.PurePath, *args, **kwargs) → ksconf.layer.LayerFile
ksconf.layer.path_in_layer(layer: pathlib.Path, path: pathlib.Path) → pathlib.Path

Check to see if path exist within layer. Returns either None, or the path without the shared prefix with layer.

ksconf.package module

class ksconf.package.AppPackager(src_path, app_name: str, output: TextIO)

Bases: object

block_local(report=True)
blocklist(patterns)
check()

Run safety checks prior to building archive:

  1. Set app name based on app.conf [package] id, if set. Otherwise confirm that the package id and top-level folder names align.
  2. Check for files or directories starting with ., makes AppInspect very grumpy!
cleanup()
combine(src, filters, layer_method='dir.d', allow_symlink=False)
expand_new_only(value: str) → Optional[str]

Expand a variable but return False if no substitution occurred

Parameters:value (str) – String that may contain {{variable}} substitution.
Returns:Expanded value if variables were expanded, else False
Return type:str
expand_var(value: str) → str

Expand a variable, if present

Parameters:value (str) – String that main contain {{variable}} substitution.
Returns:Expanded value
Return type:str
make_archive(filename)

Create a compressed tarball of the build directory.

make_manifest(calculate_hash=True) → ksconf.app.manifest.AppManifest

Create a manifest of the app’s contents.

merge_local()

Find everything in local, if it has a corresponding file in default, merge.

require_active_context()

Decorator to mark member functions that cannot be used until the context manager has been activated.

update_app_conf(version: str = None, build: str = None)

Update version and/or build in apps.conf

class ksconf.package.AppVarMagic(src_dir, build_dir, meta=None)

Bases: object

A lazy loading dict-like object to fetch things like app version and such on demand.

expand(value: str) → str

A simple Jinja2 like {{VAR}} substitution mechanism.

get_app_id()

Splunk app package id from app.conf

get_build()

Splunk app build fetched from app.conf

get_git_head()

Git HEAD rev abbreviated

get_git_last_rev()

Git abbreviated rev of the last change of the app. This may not be the same as HEAD.

get_git_tag()

Git version tag using the git describe --tags command

get_layers_hash()

Build a unique hash representing the combination of ksconf layers used.

get_layers_list()

List of ksconf layers used.

get_version()

Splunk app version fetched from app.conf

git_single_line(*args)
list_vars()

Return a list of (variable, description) available in this class.

exception ksconf.package.AppVarMagicException

Bases: KeyError

exception ksconf.package.PackagingException

Bases: Exception

ksconf.package.find_conf_in_layers(app_dir, conf, *layers)
ksconf.package.get_merged_conf(app_dir, conf, *layers)
ksconf.package.normalize_directory_mtime(path)

Walk a tree and update the directory modification times to match the newest time of the children. This results in a more predictable behavior over multiple executions.

ksconf.setup_entrypoints module

Defines all command prompt entry points for CLI actions

This is a silly hack allows for fallback mechanism when
  1. running unit tests (can happen before install)
  2. if entrypoints or pkg_resources are not available at run time (Splunk’s embedded python)
class ksconf.setup_entrypoints.Ep(name, module_name, object_name)

Bases: tuple

module_name

Alias for field number 1

name

Alias for field number 0

object_name

Alias for field number 2

class ksconf.setup_entrypoints.LocalEntryPoint(data)

Bases: object

Bare minimum stand-in for entrypoints.EntryPoint

load()
ksconf.setup_entrypoints.debug()
ksconf.setup_entrypoints.get_entrypoints_fallback(group)
ksconf.setup_entrypoints.get_entrypoints_setup()

ksconf.xmlformat module

class ksconf.xmlformat.FileReadlinesCache

Bases: object

Silly workaround for CDATA detection…

static convert_filename(filename)
readlines(filename)
class ksconf.xmlformat.SplunkSimpleXmlFormatter

Bases: object

static cdata_tags(elem: Any, tags: List[str])

Expand text to CDATA, if it isn’t already.

classmethod expand_tags(elem: Any, tags: set)

Keep <elem></elem> instead of shortening to <elem/>

classmethod format_json(elem: Any, indent=2)

Format JSON data within a Dashboard Studio dashboard. This is still pretty limited (for example, long searches still show up on a single line), but this give you at least a fighting change to figure out what’s different.

classmethod format_xml(src, dest, default_indent=2)
static guess_indent(elem: Any, default=2)
classmethod indent_tree(elem: Any, level=0, indent=2)
keep_tags = {'default', 'earliest', 'fieldset', 'label', 'latest', 'option', 'search', 'set'}

Module contents

ksconf - Ksconf Splunk CONFig tool

Design goals:

  • Multi-purpose go-to .conf tool.
  • Dependability
  • Simplicity
  • No eternal dependencies (single source file, if possible; or packable as single file.)
  • Stable CLI
  • Good scripting interface for deployment scripts and/or git hooks
exception ksconf.KsconfPluginWarning

Bases: Warning