ksconf package¶
Subpackages¶
- ksconf.app package
- ksconf.builder package
- ksconf.commands package
- Submodules
- ksconf.commands.check module
- ksconf.commands.combine module
- ksconf.commands.diff module
- ksconf.commands.filter module
- ksconf.commands.merge module
- ksconf.commands.minimize module
- ksconf.commands.package module
- ksconf.commands.promote module
- ksconf.commands.restexport module
- ksconf.commands.restpublish module
- ksconf.commands.snapshot module
- ksconf.commands.sort module
- ksconf.commands.unarchive module
- ksconf.commands.xmlformat module
- Module contents
- ksconf.conf package
- ksconf.util package
- ksconf.vc package
Submodules¶
ksconf.archive module¶
-
class
ksconf.archive.
GenArchFile
(path, mode, size, payload)¶ Bases:
tuple
-
mode
¶ Alias for field number 1
-
path
¶ Alias for field number 0
-
payload
¶ Alias for field number 3
-
size
¶ Alias for field number 2
-
-
ksconf.archive.
extract_archive
(archive_name, extract_filter: callable = None) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
gaf_filter_name_like
(pattern)¶
-
ksconf.archive.
gen_arch_file_remapper
(iterable: Iterable[ksconf.archive.GenArchFile], mapping: Sequence[Tuple[str, str]]) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
sanity_checker
(iterable: Iterable[ksconf.archive.GenArchFile]) → Iterable[ksconf.archive.GenArchFile]¶
ksconf.cli module¶
KSCONF - Ksconf Splunk CONFig tool
Optionally supports argcomplete for commandline argument (tab) completion.
Install & register with:
pip install argcomplete activate-global-python-argcomplete (in ~/.bashrc)
-
ksconf.cli.
build_cli_parser
(do_formatter=False)¶
-
ksconf.cli.
check_py
()¶
-
ksconf.cli.
check_py_sane
()¶ Run a simple python environment sanity check. Here’s the scenario, if Splunk’s python is called but not all the correct environment variables have been set, then ksconf can fail in unclear ways.
-
ksconf.cli.
cli
(argv=None, _unittest=False)¶
-
ksconf.cli.
handle_cmd_failed
(subparser, ep)¶ Build a bogus subparser for a cmd that can’t be loaded, with the only purpose of providing a more consistent user experience.
ksconf.combine module¶
-
class
ksconf.combine.
LayerCombiner
(follow_symlink: bool = False, banner: str = '', dry_run: bool = False, quiet: bool = False)¶ Bases:
object
Class to recursively combine layers (directories) into a single rendered output target directory. This is heavily used by the
ksconf combine
command as well as by thepackage
command.Typical class use case:
- ::
lc = LayerCombiner()
- # Setup source, either
- lc.set_source_dirs() OR
- lc.set_layer_root()
Call hierarch:
lc.combine() Entry point -> prepare() Directory, layer prep -> prepare_target_dir() Make dir; subclass handles marker here (combine CLI) -> pre_combine_inventory() Hook for pre-processing (or alerting) the set of files to combine -> combine_files() Main worker function -> post_combine() Optional, cleanup leftover files
-
add_layer_filter
(action, pattern)¶
-
combine
(target: pathlib.Path, *, hook_label='')¶ Combine layers into
target
directory.
-
combine_files
(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile])¶
-
conf_file_re
= re.compile('([a-z_-]+\\.conf|(default|local)\\.meta)$')¶
-
debug
(message)¶
-
filetype_handlers
= [(<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_merge_conf_files>), (<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_spec_concatenate>)]¶
-
log
(message)¶
-
post_combine
(target)¶ Hook point for post-processing after all copy/merge operations have been completed.
-
pre_combine_inventory
(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile]) → List[ksconf.layer.LayerFile]¶ Hook point for pre-processing before any files are copied/merged
-
prepare
(target: pathlib.Path)¶ Start the combine process. This includes directory checking, applying layer filtering, and marker file handling.
-
prepare_target_dir
(target: pathlib.Path)¶ Hook to ensure destination directory is ready for use. This can be overridden to adder marker file handling for use cases that need it (e.g., the ‘combine’ command)
-
classmethod
register_handler
(regex_match)¶ Decorator that registers a new file type handler. The handler is used if a file name matches a regex. Regex ‘search’ mode is used.
-
set_layer_root
(root: ksconf.layer.LayerRootBase.Layer)¶
-
set_source_dirs
(sources: List[pathlib.Path])¶
-
spec_file_re
= re.compile('\\.conf\\.spec$')¶
-
exception
ksconf.combine.
LayerCombinerException
¶ Bases:
Exception
-
ksconf.combine.
handle_merge_conf_files
(combiner: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)¶ Handle merging two or more
.conf
files.
-
ksconf.combine.
handle_spec_concatenate
(combiner: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)¶ Concatenate multiple
.spec
files. Likely aREADME.d
situation.
ksconf.compat module¶
Silly simple Python version compatibility items
-
ksconf.compat.
cache
(user_function)¶
-
ksconf.compat.
handle_py3_kw_only_args
(kw, *default_args)¶ Fake support for Python 3.8+ style keyword-only style arguments, or
*
arg syntax.Example Python 3.8+ syntax:
def f(arg, *args, a=True, b=False): ...
Example Python 3.7 (and earlier) syntax with this helper function:
def f(arg, *args, **kw_only): a, b = handle_py3_kw_only_args(kw_only, ("a", True), ("b", False)) ...
Parameters: - kw (dict) – keyword arguments provided to the calling function. Be aware that this dict will be empty after this function is done.
- default_args (tuple) – pairs of keyword argument to the caller function in argument (arg_name, default_value) order.
Raises: TypeError – if
kw
contains any keys not defined inargs
This mirrors Python’s native behavior when an unexpected argument is passed to a function.
ksconf.consts module¶
ksconf.filter module¶
-
class
ksconf.filter.
FilteredList
(flags=0, default=True)¶ Bases:
object
-
IGNORECASE
= 1¶
-
INVERT
= 2¶
-
VERBOSE
= 4¶
-
feed
(item, filter=None)¶
-
feedall
(iterable, filter=None)¶
-
has_rules
¶
-
match
(item)¶
-
match_path
(path)¶
-
match_stanza
(stanza)¶ Same as match(), but handle GLOBAL_STANZA gracefully.
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListRegex
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Regular Expression support
-
calc_regex_flags
()¶
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListSplunkGlob
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Classic wildcard support (‘*’ and ?’) plus ‘…’ or ‘**’ for multiple-path components with some (non-advertised) pass-through regex behavior
-
class
ksconf.filter.
FilteredListString
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Handle simple string comparisons
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListWildcard
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Wildcard support (handling ‘*’ and ?’) Technically fnmatch also supports [] and [!] character ranges, but we don’t advertise that
-
ksconf.filter.
create_filtered_list
(match_mode, flags=0, default=True)¶
ksconf.hook module¶
-
ksconf.hook.
get_plugin_manager
() → pluggy._manager.PluginManager¶ Return the shared pluggy PluginManager (singleton) instance.
ksconf.hookspec module¶
This module contains all the plugin definitions (or hook “specifications”) for various customization or integration points with ksconf. Not all of these have been fully tested so please let us know if something is not working as expected, or if additional arguments are needed.
Certainly the list of supported hooks will grow over time.
-
ksconf.hookspec.
ksconf_cli_init
()¶ Simple hook that is run before CLI initialization. This can be use to modify the runtime environment.
This can be used to register additional handlers, such as:
ksconf.combine.register_handler()
- Add a combination file handler (based on pattern matching)ksconf.layer.register_file_handler()
- Add file handlers for layer processing for template processing
-
ksconf.hookspec.
ksconf_cli_modify_argparse
(parser: Any, name: str)¶ Manipulate argparse rules. This could be used to add additional CLI options for other hook-added features added features
Note that this hook is called for both the top-level argparse instance as well as each subparser. The
name
argument should be inspected to determine if the parse instances is the parent (top-level) parser, or some other named subcommands.
-
ksconf.hookspec.
ksconf_cli_process_args
(args: Any)¶ Hook to capture any custom arguments added to the CLI by the
ksconf_cli_modify_argparse()
hook.
-
ksconf.hookspec.
modify_jinja_env
(env: Any)¶ Modify the Jinja2 environment object in place. This can be used to add custom filters or tests, for example. Invoked by
ksconf.layer.LayerFile_Jinja2
immediately after initial Environment creation.env
should be mutated in place.
-
ksconf.hookspec.
package_pre_archive
(app_dir: pathlib.Path, app_name: str)¶ During a
ksconf package
process, this hook executes right before the final archive is created.
-
ksconf.hookspec.
post_combine
(target: pathlib.Path, usage: str)¶ usage
should be either “combine” or “package” depending on which ksconf command was invoked. Internally the combine process is used by both.
ksconf.layer module¶
-
class
ksconf.layer.
DirectLayerRoot
(context: Optional[ksconf.layer.LayerContext] = None)¶ Bases:
ksconf.layer.LayerRootBase
A very simple direct LayerRoot implementation that relies on all layer paths to be explicitly given without any automatic detection mechanisms. You can think of this as the legacy implementation.
-
add_layer
(path: pathlib.Path)¶
-
order_layers
()¶
-
-
class
ksconf.layer.
DotDLayerRoot
(context=None)¶ Bases:
ksconf.layer.LayerRootBase
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, context: ksconf.layer.LayerContext, file_factory: Callable, prune_points: Optional[Set[pathlib.Path]] = None)¶ Bases:
ksconf.layer.Layer
-
prune_points
¶
-
walk
() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]¶
-
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter)¶ Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
-
layer_regex
= re.compile('(?P<layer>\\d\\d-[\\w_.-]+)')¶
-
list_layers
() → List[Layer]¶
-
mount_regex
= re.compile('(?P<realname>[\\w_.-]+)\\.d$')¶
-
order_layers
()¶
-
set_root
(root: pathlib.Path, follow_symlinks=None)¶ Set a root path, and auto discover all ‘.d’ directories.
Note: We currently only support
.d/<layer>
directories, a file likedefault.d/10-props.conf
won’t be handled here. A valid name would bedefault.d/10-name/props.conf
.
-
class
-
class
ksconf.layer.
FileFactory
¶ Bases:
object
-
disable
(name)¶
-
enable
(name, _enabled=True)¶
-
list_available_handlers
() → List[str]¶
-
register_handler
(name: str, **kwargs)¶
-
-
class
ksconf.layer.
LayerContext
(follow_symlink: 'bool' = False, block_files: 'Match' = re.compile('\.(bak|swp)$'), block_dirs: 'set' = <factory>, template_variables: 'dict' = <factory>)¶ Bases:
object
-
block_files
= re.compile('\\.(bak|swp)$')¶
-
follow_symlink
= False¶
-
-
exception
ksconf.layer.
LayerException
¶ Bases:
Exception
-
class
ksconf.layer.
LayerFile
(layer: ksconf.layer.LayerRootBase.Layer, relative_path: pathlib.PurePath, stat: Optional[os.stat_result] = None)¶ Bases:
os.PathLike
Abstraction of a file within a Layer
Path definitions
logical_path
- Conceptual file path. This is the final path after all layers are resolved. Think of this as the ‘destination’ file.
physical_path
- Actual file path. The location of the physical file found within a source layer. Most of the time this is the ‘source’ file, however this doesn’t take into considerations layer combining or template expansion requirements. (In the case of a template, this would be the template file)
resource_path
- Content location. Often this the
physical_path
, but in the case of abstracted layers (like templates, or archived layers), this would be the location of a temporary resource that contains the expanded/rendered content.
-
layer
¶
-
logical_path
¶
-
static
match
(path: pathlib.PurePath)¶
-
mtime
¶
-
physical_path
¶
-
relative_path
¶
-
resource_path
¶
-
size
¶
-
stat
¶
-
class
ksconf.layer.
LayerFile_Jinja2
(*args, **kwargs)¶ Bases:
ksconf.layer.LayerRenderedFile
-
jinja2_env
¶
-
static
match
(path: pathlib.PurePath)¶
-
render
(template_path: pathlib.Path) → str¶
-
static
transform_name
(path: pathlib.PurePath)¶
-
-
class
ksconf.layer.
LayerFilter
¶ Bases:
object
-
add_rule
(action, pattern)¶
-
evaluate
(layer: ksconf.layer.LayerRootBase.Layer) → bool¶
-
-
class
ksconf.layer.
LayerRenderedFile
(*args, **kwargs)¶ Bases:
ksconf.layer.LayerFile
Abstract LayerFile for rendered scenarios, such as template scenarios. A subclass really only needs to implement
match()
render()
-
logical_path
¶
-
physical_path
¶
-
render
(template_path: pathlib.Path) → str¶
-
resource_path
¶
-
static
transform_name
(path: pathlib.PurePath)¶
-
use_secure_delete
= False¶
-
-
class
ksconf.layer.
LayerRootBase
(context: Optional[ksconf.layer.LayerContext] = None)¶ Bases:
object
All ‘path’s here are relative to the ROOT.
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, context: ksconf.layer.LayerContext, file_factory: Callable)¶ Bases:
object
Basic layer Container: Connects logical and physical paths.
-
context
¶
-
get_file
(path: pathlib.Path) → ksconf.layer.LayerFile¶ Return file object (by logical path), if it exists in this layer.
-
iter_files
() → Iterator[ksconf.layer.LayerFile]¶
-
list_files
() → List[ksconf.layer.LayerFile]¶
-
logical_path
¶
-
name
¶
-
physical_path
¶
-
root
¶
-
walk
() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]¶
-
-
add_layer
(layer: Layer, do_sort=True)¶
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter) → bool¶ Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
-
get_file
(path) → Iterator[ksconf.layer.LayerFile]¶ return all layers associated with the given relative path.
-
get_layers_by_name
(name: str) → Iterator[ksconf.layer.LayerRootBase.Layer]¶
-
iter_all_files
() → Iterator[ksconf.layer.LayerFile]¶ Iterator over all physical files.
-
list_files
() → List[ksconf.layer.LayerFile]¶ Return a list of logical paths.
-
list_layer_names
() → List[str]¶
-
list_layers
() → List[Layer]¶
-
list_logical_files
() → List[ksconf.layer.LayerFile]¶ Return a list of logical paths.
-
list_physical_files
() → List[ksconf.layer.LayerFile]¶
-
order_layers
()¶
-
class
-
exception
ksconf.layer.
LayerUsageException
¶ Bases:
ksconf.layer.LayerException
ksconf.package module¶
-
class
ksconf.package.
AppPackager
(src_path, app_name: str, output: TextIO, template_variables: Optional[dict] = None, predictable_mtime: bool = True)¶ Bases:
object
-
block_local
(report=True)¶
-
blocklist
(patterns)¶
-
check
()¶ Run safety checks prior to building archive:
- Set app name based on app.conf [package] id, if set. Otherwise confirm that the package id and top-level folder names align.
- Check for files or directories starting with
.
, makes AppInspect very grumpy!
-
cleanup
()¶
-
combine
(src, filters, layer_method='dir.d', allow_symlink=False)¶
-
expand_new_only
(value: str) → Optional[str]¶ Expand a variable but return False if no substitution occurred
Parameters: value (str) – String that may contain {{variable}}
substitution.Returns: Expanded value if variables were expanded, else False Return type: str
-
expand_var
(value: str) → str¶ Expand a variable, if present
Parameters: value (str) – String that main contain {{variable}}
substitution.Returns: Expanded value Return type: str
-
freeze
(caller_name)¶
-
make_archive
(filename, temp_suffix: str = '.tmp')¶ Create a compressed tarball of the build directory.
-
make_manifest
(calculate_hash=True) → ksconf.app.manifest.AppManifest¶ Create a manifest of the app’s contents.
-
merge_local
()¶ Find everything in local, if it has a corresponding file in default, merge.
-
require_active_context
(mutable=True)¶ Decorator to mark member functions that cannot be used until the context manager has been activated.
-
update_app_conf
(version: str = None, build: str = None)¶ Update version and/or build in
apps.conf
-
-
class
ksconf.package.
AppVarMagic
(src_dir, build_dir, meta=None)¶ Bases:
object
A lazy loading dict-like object to fetch things like app version and such on demand.
-
expand
(value: str) → str¶ A simple Jinja2 like
{{VAR}}
substitution mechanism.
-
get_app_id
()¶ Splunk app package id from app.conf
-
get_build
()¶ Splunk app build fetched from app.conf
-
get_git_head
()¶ Git HEAD rev abbreviated
-
get_git_last_rev
()¶ Git abbreviated rev of the last change of the app. This may not be the same as HEAD.
-
get_git_tag
()¶ Git version tag using the
git describe --tags
command
-
get_layers_hash
()¶ Build a unique hash representing the combination of ksconf layers used.
-
get_layers_list
()¶ List of ksconf layers used.
-
get_version
()¶ Splunk app version fetched from app.conf
-
git_single_line
(*args)¶
-
list_vars
()¶ Return a list of (variable, description) available in this class.
-
-
exception
ksconf.package.
AppVarMagicException
¶ Bases:
KeyError
-
exception
ksconf.package.
PackagingException
¶ Bases:
Exception
-
ksconf.package.
find_conf_in_layers
(app_dir, conf, *layers)¶
-
ksconf.package.
get_merged_conf
(app_dir, conf, *layers)¶
-
ksconf.package.
normalize_directory_mtime
(path)¶ Walk a tree and update the directory modification times to match the newest time of the children. This results in a more predictable behavior over multiple executions.
ksconf.setup_entrypoints module¶
Defines all command prompt entry points for CLI actions
- This is a silly hack allows for fallback mechanism when
- running unit tests (can happen before install)
- if entrypoints or pkg_resources are not available at run time (Splunk’s embedded python)
-
class
ksconf.setup_entrypoints.
Ep
(name, module_name, object_name)¶ Bases:
tuple
-
module_name
¶ Alias for field number 1
-
name
¶ Alias for field number 0
-
object_name
¶ Alias for field number 2
-
-
class
ksconf.setup_entrypoints.
LocalEntryPoint
(data)¶ Bases:
object
Bare minimum stand-in for entrypoints.EntryPoint
-
load
()¶
-
-
ksconf.setup_entrypoints.
debug
()¶
-
ksconf.setup_entrypoints.
get_entrypoints_fallback
(group)¶
-
ksconf.setup_entrypoints.
get_entrypoints_setup
()¶
ksconf.xmlformat module¶
-
class
ksconf.xmlformat.
FileReadlinesCache
¶ Bases:
object
Silly workaround for CDATA detection…
-
static
convert_filename
(filename)¶
-
readlines
(filename)¶
-
static
-
class
ksconf.xmlformat.
SplunkSimpleXmlFormatter
¶ Bases:
object
Expand text to CDATA, if it isn’t already.
Keep <elem></elem> instead of shortening to <elem/>
-
classmethod
format_json
(elem: Any, indent=2)¶ Format JSON data within a Dashboard Studio dashboard. This is still pretty limited (for example, long searches still show up on a single line), but this give you at least a fighting change to figure out what’s different.
-
classmethod
format_xml
(src, dest, default_indent=2)¶
-
static
guess_indent
(elem: Any, default=2)¶
-
classmethod
indent_tree
(elem: Any, level=0, indent=2)¶
Module contents¶
ksconf - Ksconf Splunk CONFig tool
Design goals:
- Multi-purpose go-to
.conf
tool.- Dependability
- Simplicity
- No eternal dependencies (single source file, if possible; or packable as single file.)
- Stable CLI
- Good scripting interface for deployment scripts and/or git hooks
-
exception
ksconf.
KsconfPluginWarning
¶ Bases:
Warning