ksconf package¶
Subpackages¶
- ksconf.app package
- ksconf.builder package
- ksconf.commands package
- Submodules
- ksconf.commands.check module
- ksconf.commands.combine module
- ksconf.commands.diff module
- ksconf.commands.filter module
- ksconf.commands.merge module
- ksconf.commands.minimize module
- ksconf.commands.package module
- ksconf.commands.promote module
- ksconf.commands.restexport module
- ksconf.commands.restpublish module
- ksconf.commands.snapshot module
- ksconf.commands.sort module
- ksconf.commands.unarchive module
- ksconf.commands.xmlformat module
- Module contents
- ksconf.conf package
- ksconf.util package
- ksconf.vc package
Submodules¶
ksconf.archive module¶
-
class
ksconf.archive.
GenArchFile
(path, mode, size, payload)¶ Bases:
tuple
-
mode
¶ Alias for field number 1
-
path
¶ Alias for field number 0
-
payload
¶ Alias for field number 3
-
size
¶ Alias for field number 2
-
-
ksconf.archive.
extract_archive
(archive_name, extract_filter: callable = None) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
gaf_filter_name_like
(pattern)¶
-
ksconf.archive.
gen_arch_file_remapper
(iterable: Iterable[ksconf.archive.GenArchFile], mapping: Sequence[Tuple[str, str]]) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
sanity_checker
(iterable: Iterable[ksconf.archive.GenArchFile]) → Iterable[ksconf.archive.GenArchFile]¶
ksconf.cli module¶
KSCONF - Ksconf Splunk CONFig tool
Optionally supports argcomplete for commandline argument (tab) completion.
Install & register with:
pip install argcomplete activate-global-python-argcomplete (in ~/.bashrc)
-
ksconf.cli.
build_cli_parser
(do_formatter=False)¶
-
ksconf.cli.
check_py
()¶
-
ksconf.cli.
check_py_sane
()¶ Run a simple python environment sanity check. Here’s the scenario, if Splunk’s python is called but not all the correct environment variables have been set, then ksconf can fail in unclear ways.
-
ksconf.cli.
cli
(argv=None, _unittest=False)¶
-
ksconf.cli.
handle_cmd_failed
(subparser, ep)¶ Build a bogus subparser for a cmd that can’t be loaded, with the only purpose of providing a more consistent user experience.
ksconf.combine module¶
-
class
ksconf.combine.
LayerCombiner
(follow_symlink: bool = False, banner: str = '', dry_run: bool = False, quiet: bool = False)¶ Bases:
object
Class to recursively combine layers (directories) into a single rendered output target directory. This is heavily used by the
ksconf combine
command as well as by thepackage
command.Typical class use case:
- ::
lc = LayerCombiner()
- # Setup source, either
- lc.set_source_dirs() OR
- lc.set_layer_root()
Call hierarch:
lc.combine() Entry point -> prepare() Directory, layer prep -> prepare_target_dir() Make dir; subclass handles marker here (combine CLI) -> pre_combine_inventory() Hook for pre-processing (or alerting) the set of files to combine -> combine_files() Main worker function -> post_combine() Optional, cleanup leftover files
-
add_layer_filter
(action, pattern)¶
-
combine
(target: pathlib.Path, *, hook_label='')¶ Combine layers into
target
directory. Anyhook_label
given will be passed to the plugin system via theusage
field.
-
combine_files
(target: pathlib.Path, src_files: list)¶
-
conf_file_re
= re.compile('([a-z_-]+\\.conf|(default|local)\\.meta)$')¶
-
debug
(message)¶
-
filetype_handlers
= [(<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_merge_conf_files>), (<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_spec_concatenate>)]¶
-
log
(message)¶
-
post_combine
(target)¶ Hook point for post-processing after all copy/merge operations have been completed.
-
pre_combine_inventory
(target: pathlib.Path, src_files: list) → list¶ Hook point for pre-processing before any files are copied/merged
-
prepare
(target: pathlib.Path)¶ Start the combine process. This includes directory checking, applying layer filtering, and marker file handling.
-
prepare_target_dir
(target: pathlib.Path)¶ Hook to ensure destination directory is ready for use. This can be overridden to adder marker file handling for use cases that need it (e.g., the ‘combine’ command)
-
classmethod
register_handler
(regex_match)¶ Decorator that registers a new file type handler. The handler is used if a file name matches a regex. Regex ‘search’ mode is used.
-
set_layer_root
(root: ksconf.layer.LayerRootBase.Layer)¶
-
set_source_dirs
(sources: list)¶
-
spec_file_re
= re.compile('\\.conf\\.spec$')¶
-
exception
ksconf.combine.
LayerCombinerException
¶ Bases:
Exception
-
ksconf.combine.
handle_merge_conf_files
(combiner: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: list, dry_run)¶ Handle merging two or more
.conf
files.
-
ksconf.combine.
handle_spec_concatenate
(combiner: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: list, dry_run)¶ Concatenate multiple
.spec
files. Likely aREADME.d
situation.
ksconf.compat module¶
Silly simple Python version compatibility items
-
ksconf.compat.
Dict
¶ alias of
builtins.dict
-
ksconf.compat.
List
¶ alias of
builtins.list
-
ksconf.compat.
Set
¶ alias of
builtins.set
-
ksconf.compat.
Tuple
¶ alias of
builtins.tuple
-
ksconf.compat.
cache
(user_function)¶ Simple lightweight unbounded cache. Sometimes called “memoize”.
ksconf.consts module¶
-
class
ksconf.consts.
SmartEnum
(*args, **kwds)¶ Bases:
enum.Enum
-
CREATE
= 'created'¶
-
NOCHANGE
= 'unchanged'¶
-
UPDATE
= 'updated'¶
-
-
ksconf.consts.
is_debug
()¶
ksconf.filter module¶
-
class
ksconf.filter.
FilteredList
(flags=0, default=True)¶ Bases:
object
-
IGNORECASE
= 1¶
-
INVERT
= 2¶
-
VERBOSE
= 4¶
-
feed
(item, filter=None)¶
-
feedall
(iterable, filter=None)¶
-
has_rules
¶
-
match
(item)¶
-
match_path
(path)¶
-
match_stanza
(stanza)¶ Same as match(), but handle GLOBAL_STANZA gracefully.
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListRegex
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Regular Expression support
-
calc_regex_flags
()¶
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListSplunkGlob
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Classic wildcard support (‘*’ and ?’) plus ‘…’ or ‘**’ for multiple-path components with some (non-advertised) pass-through regex behavior
-
class
ksconf.filter.
FilteredListString
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Handle simple string comparisons
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListWildcard
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Wildcard support (handling ‘*’ and ?’) Technically fnmatch also supports [] and [!] character ranges, but we don’t advertise that
-
ksconf.filter.
create_filtered_list
(match_mode, flags=0, default=True)¶
ksconf.hook module¶
-
ksconf.hook.
get_plugin_manager
() → ksconf.hook._plugin_manager¶ Return the shared pluggy PluginManager (singleton) instance.
This is for backwards compatibility. This was only added in v0.11.6; and replaced immediately after.
ksconf.hookspec module¶
This module contains all the plugin definitions (or hook “specifications”) for various customization or integration points with ksconf. Not all of these have been fully tested so please let us know if something is not working as expected, or if additional arguments are needed.
See ksconf plugins on pypi for a list of currently available plugins.
-
class
ksconf.hookspec.
KsconfHookSpecs
(*args, **kwargs)¶ Bases:
typing.Protocol
Ksconf plugin specifications for all known supported functions.
Grouping these functions together in a single class allows for type support it supports typing. This adds a level of validation to the code base where a hook is invoked via
plugin_manger.hook.<hook_name>()
.If you are implementing one of these hooks, please note that you can simple make top-level function, no need to implement a class.
-
static
ksconf_cli_init
()¶ Simple hook that is run before CLI initialization. This can be use to modify the runtime environment.
This can be used to register additional handlers, such as:
ksconf.combine.register_handler()
- Add a combination file handler. File types are limited to pattern matching.ksconf.layer.register_file_handler()
- Add file handlers for layer processing for template processing
-
static
ksconf_cli_modify_argparse
(parser: Any, name: str)¶ Manipulate argparse rules. This could be used to add additional CLI options for other hook-added features added features
Note that this hook is called for both the top-level argparse instance as well as each subparser. The
name
argument should be inspected to determine if the parse instances is the parent (top-level) parser, or some other named subcommands.
-
static
ksconf_cli_process_args
(args: Any)¶ Hook to capture all parsed arguments, includes any custom arguments added to the CLI via the the
ksconf_cli_modify_argparse()
hook.args
can be mutated directly, if needed.
-
static
modify_jinja_env
(env: Any)¶ Modify the Jinja2 environment object. This can be used to add custom filters or tests, for example.
Invoked by
LayerFile_Jinja2
immediately after initial Environment creation.env
should be mutated in place.
-
static
package_pre_archive
(app_dir: pathlib.Path, app_name: str)¶ Modify, inventory, or test the contents of an app before the final packaging commands. This can be triggered from the
ksconf package
command or via the API.During a
ksconf package
process, this hook executes right before the final archive is created. All local merging, app version or build updates, and so on are completed before this hook is executed.From an API perspective, this hook is called from
ksconf.package.AppPackager
whenever a content freeze occurs, which is typically whenmake_archive()
ormake_manifest()
is invoked.
-
static
post_combine
(target: pathlib.Path, usage: str)¶ Trigger a custom action after a layer combining operation. This is used by multiple ksconf subcommands and the API.
This trigger could be used to modify the file system, trigger external operations, track/audit behaviors, and so on.
When using CLI commands,
usage
should be either “combine” or “package” depending on which ksconf command was invoked. Direct invocation ofLayerCombiner
can pass along a custom usage label and avoid impacting CLI, when desirable.If your goal is to only trigger an action during the app packaging process, also consider the
package_pre_archive()
hook, which may be more appropriate.
-
static
ksconf.layer module¶
-
class
ksconf.layer.
DirectLayerRoot
(context: ksconf.layer.LayerContext = None)¶ Bases:
ksconf.layer.LayerRootBase
A very simple direct LayerRoot implementation that relies on all layer paths to be explicitly given without any automatic detection mechanisms. You can think of this as the legacy implementation.
-
add_layer
(path: pathlib.Path)¶
-
order_layers
()¶
-
-
class
ksconf.layer.
DotDLayerRoot
(context=None)¶ Bases:
ksconf.layer.LayerRootBase
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, context: ksconf.layer.LayerContext, file_factory: Callable, prune_points: set = None)¶ Bases:
ksconf.layer.Layer
-
prune_points
¶
-
walk
() → Iterator[tuple]¶
-
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter)¶ Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
-
layer_regex
= re.compile('(?P<layer>\\d\\d-[\\w_.-]+)')¶
-
list_layers
() → List[Layer]¶
-
mount_regex
= re.compile('(?P<realname>[\\w_.-]+)\\.d$')¶
-
order_layers
()¶
-
set_root
(root: pathlib.Path, follow_symlinks=None)¶ Set a root path, and auto discover all ‘.d’ directories.
Note: We currently only support
.d/<layer>
directories, a file likedefault.d/10-props.conf
won’t be handled here. A valid name would bedefault.d/10-name/props.conf
.
-
class
-
class
ksconf.layer.
FileFactory
¶ Bases:
object
-
disable
(name)¶
-
enable
(name, _enabled=True)¶
-
list_available_handlers
() → list¶
-
register_handler
(name: str, **kwargs)¶
-
-
class
ksconf.layer.
LayerContext
(follow_symlink: 'bool' = False, block_files: 'Match' = re.compile('\.(bak|swp)$'), block_dirs: 'set' = <factory>, template_variables: 'dict' = <factory>)¶ Bases:
object
-
block_files
= re.compile('\\.(bak|swp)$')¶
-
follow_symlink
= False¶
-
-
exception
ksconf.layer.
LayerException
¶ Bases:
Exception
-
class
ksconf.layer.
LayerFile
(layer: ksconf.layer.LayerRootBase.Layer, relative_path: pathlib.PurePath, stat: os.stat_result = None)¶ Bases:
os.PathLike
Abstraction of a file within a Layer
Path definitions
logical_path
- Conceptual file path. This is the final path after all layers are resolved. Think of this as the ‘destination’ file.
physical_path
- Actual file path. The location of the physical file found within a source layer. Most of the time this is the ‘source’ file, however this doesn’t take into considerations layer combining or template expansion requirements. (In the case of a template, this would be the template file)
resource_path
- Content location. Often this the
physical_path
, but in the case of abstracted layers (like templates, or archived layers), this would be the location of a temporary resource that contains the expanded/rendered content.
-
layer
¶
-
logical_path
¶
-
static
match
(path: pathlib.PurePath)¶
-
mtime
¶
-
physical_path
¶
-
relative_path
¶
-
resource_path
¶
-
size
¶
-
stat
¶
-
class
ksconf.layer.
LayerFile_Jinja2
(*args, **kwargs)¶ Bases:
ksconf.layer.LayerRenderedFile
-
jinja2_env
¶
-
static
match
(path: pathlib.PurePath)¶
-
render
(template_path: pathlib.Path) → str¶
-
static
transform_name
(path: pathlib.PurePath)¶
-
-
class
ksconf.layer.
LayerFilter
¶ Bases:
object
-
add_rule
(action, pattern)¶
-
evaluate
(layer: ksconf.layer.LayerRootBase.Layer) → bool¶
-
-
class
ksconf.layer.
LayerRenderedFile
(*args, **kwargs)¶ Bases:
ksconf.layer.LayerFile
Abstract LayerFile for rendered scenarios, such as template scenarios. A subclass really only needs to implement
match()
render()
-
logical_path
¶
-
physical_path
¶
-
render
(template_path: pathlib.Path) → str¶
-
resource_path
¶
-
static
transform_name
(path: pathlib.PurePath)¶
-
use_secure_delete
= False¶
-
-
class
ksconf.layer.
LayerRootBase
(context: ksconf.layer.LayerContext = None)¶ Bases:
object
All ‘path’s here are relative to the ROOT.
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, context: ksconf.layer.LayerContext, file_factory: Callable)¶ Bases:
object
Basic layer Container: Connects logical and physical paths.
-
context
¶
-
get_file
(path: pathlib.Path) → ksconf.layer.LayerFile¶ Return file object (by logical path), if it exists in this layer.
-
iter_files
() → Iterator[ksconf.layer.LayerFile]¶
-
list_files
() → list¶
-
logical_path
¶
-
name
¶
-
physical_path
¶
-
root
¶
-
walk
() → Iterator[tuple]¶
-
-
add_layer
(layer: Layer, do_sort=True)¶
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter) → bool¶ Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
-
get_file
(path) → Iterator[ksconf.layer.LayerFile]¶ return all layers associated with the given relative path.
-
get_layers_by_name
(name: str) → Iterator[ksconf.layer.LayerRootBase.Layer]¶
-
iter_all_files
() → Iterator[ksconf.layer.LayerFile]¶ Iterator over all physical files.
-
list_files
() → list¶ Return a list of logical paths.
-
list_layer_names
() → list¶
-
list_layers
() → List[Layer]¶
-
list_logical_files
() → list¶ Return a list of logical paths.
-
list_physical_files
() → list¶
-
order_layers
()¶
-
class
-
exception
ksconf.layer.
LayerUsageException
¶ Bases:
ksconf.layer.LayerException
ksconf.package module¶
-
class
ksconf.package.
AppPackager
(src_path, app_name: str, output: TextIO, template_variables: dict = None, predictable_mtime: bool = True)¶ Bases:
object
-
block_local
(report=True)¶
-
blocklist
(patterns)¶
-
check
()¶ Run safety checks prior to building archive:
- Set app name based on app.conf [package] id, if set. Otherwise confirm that the package id and top-level folder names align.
- Check for files or directories starting with
.
, makes AppInspect very grumpy!
-
cleanup
()¶
-
combine
(src, filters, layer_method='dir.d', allow_symlink=False)¶
-
expand_new_only
(value: str) → Optional[str, None]¶ Expand a variable but return False if no substitution occurred
Parameters: value (str) – String that may contain {{variable}}
substitution.Returns: Expanded value if variables were expanded, else False Return type: str
-
expand_var
(value: str) → str¶ Expand a variable, if present
Parameters: value (str) – String that main contain {{variable}}
substitution.Returns: Expanded value Return type: str
-
freeze
(caller_name)¶ Initiate a content freeze by restricting mutable methods. The “package_pre_archive” hook is invoked before freeze operation. Such hooks may choose to mutate the filesystem at
app_dir
, the only assumption is that all work is done before the hook returns.Freeze can be safely called multiple times.
caller_name
is simply a label used in an exception message if the programmer screwed up.
-
make_archive
(filename, temp_suffix: str = '.tmp')¶ Create a compressed tarball of the build directory.
-
make_manifest
(calculate_hash=True) → ksconf.app.manifest.AppManifest¶ Create a manifest of the app’s contents.
-
merge_local
()¶ Find everything in local, if it has a corresponding file in default, merge.
-
require_active_context
(mutable=True)¶ Decorator to mark member functions that cannot be used until the context manager has been activated.
-
update_app_conf
(version: str = None, build: str = None)¶ Update version and/or build in
apps.conf
-
-
class
ksconf.package.
AppVarMagic
(src_dir, build_dir, meta=None)¶ Bases:
object
A lazy loading dict-like object to fetch things like app version and such on demand.
-
expand
(value: str) → str¶ A simple Jinja2 like
{{VAR}}
substitution mechanism.
-
get_app_id
()¶ Splunk app package id from app.conf
-
get_build
()¶ Splunk app build fetched from app.conf
-
get_git_head
()¶ Git HEAD rev abbreviated
-
get_git_last_rev
()¶ Git abbreviated rev of the last change of the app. This may not be the same as HEAD.
-
get_git_tag
()¶ Git version tag using the
git describe --tags
command
-
get_layers_hash
()¶ Build a unique hash representing the combination of ksconf layers used.
-
get_layers_list
()¶ List of ksconf layers used.
-
get_version
()¶ Splunk app version fetched from app.conf
-
git_single_line
(*args)¶
-
list_vars
()¶ Return a list of (variable, description) available in this class.
-
-
exception
ksconf.package.
AppVarMagicException
¶ Bases:
KeyError
-
exception
ksconf.package.
PackagingException
¶ Bases:
Exception
-
ksconf.package.
find_conf_in_layers
(app_dir, conf, *layers)¶
-
ksconf.package.
get_merged_conf
(app_dir, conf, *layers)¶
-
ksconf.package.
normalize_directory_mtime
(path)¶ Walk a tree and update the directory modification times to match the newest time of the children. This results in a more predictable behavior over multiple executions.
ksconf.setup_entrypoints module¶
Defines all command prompt entry points for CLI actions
- This is a silly hack allows for fallback mechanism when
- running unit tests (can happen before install)
- unexpected issues with importlib.metadata or backport
-
class
ksconf.setup_entrypoints.
Ep
(name, module_name, object_name)¶ Bases:
tuple
-
module_name
¶ Alias for field number 1
-
name
¶ Alias for field number 0
-
object_name
¶ Alias for field number 2
-
-
class
ksconf.setup_entrypoints.
LocalEntryPoint
(data)¶ Bases:
object
Bare minimum stand-in for entrypoints.EntryPoint
-
load
()¶
-
-
ksconf.setup_entrypoints.
debug
()¶
-
ksconf.setup_entrypoints.
get_entrypoints_fallback
(group)¶
-
ksconf.setup_entrypoints.
get_entrypoints_setup
()¶ Build entry point text descriptions for ksconf packaging
ksconf.xmlformat module¶
-
class
ksconf.xmlformat.
FileReadlinesCache
¶ Bases:
object
Silly workaround for CDATA detection…
-
static
convert_filename
(filename)¶
-
readlines
(filename)¶
-
static
-
class
ksconf.xmlformat.
SplunkSimpleXmlFormatter
¶ Bases:
object
Expand text to CDATA, if it isn’t already.
Keep <elem></elem> instead of shortening to <elem/>
-
classmethod
format_json
(elem: Any, indent=2)¶ Format JSON data within a Dashboard Studio dashboard. This is still pretty limited (for example, long searches still show up on a single line), but this give you at least a fighting change to figure out what’s different.
-
classmethod
format_xml
(src, dest, default_indent=2)¶
-
static
guess_indent
(elem: Any, default=2)¶
-
classmethod
indent_tree
(elem: Any, level=0, indent=2)¶
Module contents¶
ksconf - Ksconf Splunk CONFig tool
Design goals:
- Multi-purpose go-to
.conf
tool.- Dependability
- Simplicity
- No eternal dependencies (single source file, if possible; or packable as single file.)
- Stable CLI
- Good scripting interface for deployment scripts and/or git hooks
-
exception
ksconf.
KsconfPluginWarning
¶ Bases:
Warning