ksconf namespace
Subpackages
- ksconf.app package
- Submodules
- ksconf.app.deploy module
- ksconf.app.facts module
AppFacts
AppFacts.allows_disable
AppFacts.author
AppFacts.build
AppFacts.check_for_updates
AppFacts.deployer_lookups_push_mode
AppFacts.deployer_push_mode
AppFacts.description
AppFacts.from_app_dir()
AppFacts.from_archive()
AppFacts.from_conf()
AppFacts.id
AppFacts.install_source_checksum
AppFacts.install_source_local_checksum
AppFacts.is_configured
AppFacts.is_visible
AppFacts.label
AppFacts.name
AppFacts.state
AppFacts.state_change_requires_restart
AppFacts.to_dict()
AppFacts.to_tiny_dict()
AppFacts.version
- ksconf.app.manifest module
AppArchiveContentError
AppArchiveError
AppManifest
AppManifestFile
AppManifestStorageError
AppManifestStorageInvalid
StoredArchiveManifest
StoredArchiveManifest.archive
StoredArchiveManifest.from_dict()
StoredArchiveManifest.from_file()
StoredArchiveManifest.from_json_manifest()
StoredArchiveManifest.hash
StoredArchiveManifest.manifest
StoredArchiveManifest.mtime
StoredArchiveManifest.read_json_manifest()
StoredArchiveManifest.size
StoredArchiveManifest.to_dict()
StoredArchiveManifest.write_json_manifest()
create_manifest_from_archive()
get_stored_manifest_name()
load_manifest_for_archive()
- Module contents
- ksconf.builder package
- Submodules
- ksconf.builder.cache module
CachedRun
CachedRun.STATE_DISABLED
CachedRun.STATE_EXISTS
CachedRun.STATE_NEW
CachedRun.STATE_TAINT
CachedRun.cache_dir
CachedRun.cached_inputs
CachedRun.cached_outputs
CachedRun.config_file
CachedRun.disable()
CachedRun.dump()
CachedRun.exists
CachedRun.inputs_identical()
CachedRun.is_disabled
CachedRun.is_expired
CachedRun.is_new
CachedRun.load()
CachedRun.rename()
CachedRun.root
CachedRun.set_cache_info()
CachedRun.set_settings()
CachedRun.taint()
FileSet
fingerprint_hash()
fingerprint_stat()
- ksconf.builder.core module
- ksconf.builder.steps module
- Module contents
- ksconf.commands namespace
- Submodules
- ksconf.commands.attr module
- ksconf.commands.check module
- ksconf.commands.combine module
- ksconf.commands.diff module
- ksconf.commands.filter module
- ksconf.commands.merge module
- ksconf.commands.minimize module
- ksconf.commands.package module
- ksconf.commands.promote module
- ksconf.commands.restexport module
- ksconf.commands.restpublish module
- ksconf.commands.snapshot module
- ksconf.commands.sort module
- ksconf.commands.unarchive module
- ksconf.commands.xmlformat module
- ksconf.conf package
- Submodules
- ksconf.conf.delta module
- ksconf.conf.merge module
- ksconf.conf.meta module
- ksconf.conf.parser module
ConfParserException
DuplicateEnum
DuplicateKeyException
DuplicateStanzaException
Token
conf_attr_boolean()
cont_handler()
detect_by_bom()
inject_section_comments()
parse_conf()
parse_conf_stream()
parse_string()
section_reader()
smart_write_conf()
splitup_kvpairs()
update_conf
write_conf()
write_conf_stream()
- Module contents
- ksconf.util package
- ksconf.vc package
Submodules
ksconf.archive module
- class ksconf.archive.GenArchFile(path, mode, size, payload)
Bases:
NamedTuple
- mode: int
Alias for field number 1
- path: str
Alias for field number 0
- payload: bytes | None
Alias for field number 3
- size: int
Alias for field number 2
- ksconf.archive.extract_archive(archive_name, extract_filter: callable = None) Iterable[GenArchFile]
- ksconf.archive.gaf_filter_name_like(pattern)
- ksconf.archive.gen_arch_file_remapper(iterable: Iterable[GenArchFile], mapping: Sequence[Tuple[str, str]]) Iterable[GenArchFile]
- ksconf.archive.sanity_checker(iterable: Iterable[GenArchFile]) Iterable[GenArchFile]
ksconf.cli module
KSCONF - Ksconf Splunk CONFig tool
Optionally supports argcomplete for commandline argument (tab) completion.
Install & register with:
pip install argcomplete activate-global-python-argcomplete (in ~/.bashrc)
- ksconf.cli.build_cli_parser(do_formatter=False)
- ksconf.cli.check_py()
- ksconf.cli.check_py_sane()
Run a simple python environment sanity check. Here’s the scenario, if Splunk’s python is called but not all the correct environment variables have been set, then ksconf can fail in unclear ways.
- ksconf.cli.cli(argv=None, _unittest=False)
- ksconf.cli.handle_cmd_failed(subparser, ep)
Build a bogus subparser for a cmd that can’t be loaded, with the only purpose of providing a more consistent user experience.
ksconf.combine module
- class ksconf.combine.LayerCombiner(follow_symlink: bool = False, banner: str = '', dry_run: bool = False, quiet: bool = False)
Bases:
object
Class to recursively combine layers (directories) into a single rendered output target directory. This is heavily used by the
ksconf combine
command as well as by thepackage
command.Typical class use case:
- ::
lc = LayerCombiner()
- # Setup source, either
lc.set_source_dirs() OR
lc.set_layer_root()
Call hierarch:
lc.combine() Entry point -> prepare() Directory, layer prep -> prepare_target_dir() Make dir; subclass handles marker here (combine CLI) -> pre_combine_inventory() Hook for pre-processing (or alerting) the set of files to combine -> combine_files() Main worker function -> post_combine() Optional, cleanup leftover files
- add_layer_filter(action, pattern)
- combine(target: Path, *, hook_label='')
Combine layers into
target
directory. Anyhook_label
given will be passed to the plugin system via theusage
field.
- conf_file_re = re.compile('([a-z_-]+\\.conf|(default|local)\\.meta)$')
- debug(message)
- filetype_handlers: list[tuple[Callable, Callable]] = [(<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_merge_conf_files>), (<function LayerCombiner.register_handler.<locals>.match_f>, <function handle_spec_concatenate>)]
- log(message)
- post_combine(target)
Hook point for post-processing after all copy/merge operations have been completed.
- pre_combine_inventory(target: Path, src_files: list[LayerFile]) list[LayerFile]
Hook point for pre-processing before any files are copied/merged
- prepare(target: Path)
Start the combine process. This includes directory checking, applying layer filtering, and marker file handling.
- prepare_target_dir(target: Path)
Hook to ensure destination directory is ready for use. This can be overridden to adder marker file handling for use cases that need it (e.g., the ‘combine’ command)
- classmethod register_handler(regex_match)
Decorator that registers a new file type handler. The handler is used if a file name matches a regex. Regex ‘search’ mode is used.
- set_source_dirs(sources: list[Path])
- spec_file_re = re.compile('\\.conf\\.spec$')
- exception ksconf.combine.LayerCombinerException
Bases:
Exception
- ksconf.combine.handle_merge_conf_files(combiner: LayerCombiner, dest_path: Path, sources: list[LayerFile], dry_run)
Handle merging two or more
.conf
files.
- ksconf.combine.handle_spec_concatenate(combiner: LayerCombiner, dest_path: Path, sources: list[LayerFile], dry_run)
Concatenate multiple
.spec
files. Likely aREADME.d
situation.
- ksconf.combine.register_handler(regex_match)
Decorator that registers a new file type handler. The handler is used if a file name matches a regex. Regex ‘search’ mode is used.
ksconf.command module
ksconf.command:
Helpers functions and classes in support of the actual commands that live under
ksconf.commands.*
.
Note that ksconf.commands
is a namespace package, which can be contributed
to by multiple python packages (technically called “distributions”). Because of
this, there can be no __init__.py, which is where this content logically belongs.
- class ksconf.command.ConfFileType(mode='r', action='open', parse_profile: Dict = None, accept_dir: bool = False)
Bases:
object
Factory for creating conf file object types; returns a lazy-loader ConfFile proxy class
Started from FileType() and then changed everything. With our use case, it’s often necessary to delay writing, or read before writing to a conf file (depending on whether or not –dry-run mode is enabled, for example.)
Instances of FileType are typically passed as type= arguments to the ArgumentParser add_argument() method.
- Parameters:
mode (str) – How the file is to be opened. Accepts “r”, “w”, and “r+”.
action (str) – Determine how much work should be handled during argument parsing vs handed off to the caller. Supports ‘none’, ‘open’, ‘load’. Full descriptions below.
parse_profile – parsing configuration settings passed along to the parser
accept_dir (bool) – Should the CLI accept a directory of config files instead of an individual file. Defaults to False.
Values for action
Action
Description
none
No preparation or testing is done on the filename.
open
Ensure the file exists and can be opened.
load
Ensure the file can be opened and parsed successfully.
Once invoked, instances of this class will return a
ConfFileProxy
object, or aConfDirProxy
object if a directory is passed in via the CLI.
- class ksconf.command.KsconfCmd(name)
Bases:
object
Ksconf command specification base class.
- add_parser(subparser)
- description = None
- exit(exit_code)
Allow overriding for unittesting or other high-level functionality, like an interactive interface.
- format = 'default'
- help = None
- launch(args)
Handle flow control between pre_run() / run() / post_run()
- maturity = 'alpha'
- parse_conf(path: str, mode: str = 'r', profile: Dict = None, raw_exec: bool = False) ConfFileProxy
- parse_extra_vars(vars: str, arg_name='argument') dict
Argument can be either a string, or a @file
- post_run(args, exec_info=None)
Optional custom clean up method. Always called if run() was. The presence of exc_info indicates failure.
- pre_run(args)
Optional pre-run hook. Any exceptions or non-0 return code, will prevent run()/post_run() from being called.
- register_args(parser: ArgumentParser)
This function in passed the
- run(args)
Actual works happens here. Return code should be an EXIT_CODE_* from consts.
- version_extra = None
- ksconf.command.add_splunkd_access_args(parser: ArgumentParser) ArgumentParser
- ksconf.command.add_splunkd_namespace(parser: ArgumentParser) ArgumentParser
- ksconf.command.dedent(text)
Remove any common leading whitespace from every line in text.
This can be used to make triple-quoted strings line up with the left edge of the display, while still presenting them in the source code in indented form.
Note that tabs and spaces are both treated as whitespace, but they are not equal: the lines ” hello” and “thello” are considered to have no common leading whitespace.
Entirely blank lines are normalized to a newline character.
- ksconf.command.get_all_ksconf_cmds(on_error='warn')
- ksconf.command.get_entrypoints(group, name=None)
ksconf.compat module
Silly simple Python version compatibility items
- ksconf.compat.Dict
alias of
dict
- ksconf.compat.List
alias of
list
- ksconf.compat.Set
alias of
set
- ksconf.compat.Tuple
alias of
tuple
- ksconf.compat.cache(user_function, /)
Simple lightweight unbounded cache. Sometimes called “memoize”.
ksconf.consts module
- class ksconf.consts.SmartEnum(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)
Bases:
Enum
- CREATE = 'created'
- NOCHANGE = 'unchanged'
- UPDATE = 'updated'
- ksconf.consts.is_debug()
ksconf.filter module
- class ksconf.filter.FilteredList(flags=0, default=True)
Bases:
object
- IGNORECASE = 1
- INVERT = 2
- VERBOSE = 4
- feed(item, filter=None)
- feedall(iterable, filter=None)
- property has_rules
- match(item)
- match_path(path)
- match_stanza(stanza)
Same as match(), but handle GLOBAL_STANZA gracefully.
- reset_counters()
- class ksconf.filter.FilteredListRegex(flags=0, default=True)
Bases:
FilteredList
Regular Expression support
- calc_regex_flags()
- reset_counters()
- class ksconf.filter.FilteredListSplunkGlob(flags=0, default=True)
Bases:
FilteredListRegex
Classic wildcard support (‘*’ and ?’) plus ‘…’ or ‘**’ for multiple-path components with some (non-advertised) pass-through regex behavior
- class ksconf.filter.FilteredListString(flags=0, default=True)
Bases:
FilteredList
Handle simple string comparisons
- reset_counters()
- class ksconf.filter.FilteredListWildcard(flags=0, default=True)
Bases:
FilteredListRegex
Wildcard support (handling ‘*’ and ?’) Technically fnmatch also supports [] and [!] character ranges, but we don’t advertise that
- ksconf.filter.create_filtered_list(match_mode: str, flags=0, default=True) FilteredList
ksconf.hook module
- exception ksconf.hook.BadPluginWarning
Bases:
UserWarning
Issue with one or more plugins
- ksconf.hook.get_plugin_manager() _plugin_manager
Return the shared pluggy PluginManager (singleton) instance.
This is for backwards compatibility. This was only added in v0.11.6; and replaced immediately after.
ksconf.hookspec module
This module contains all the plugin definitions (or hook “specifications”) for various customization or integration points with ksconf. Not all of these have been fully tested so please let us know if something is not working as expected, or if additional arguments are needed.
See ksconf plugins on pypi for a list of currently available plugins.
- class ksconf.hookspec.KsconfHookSpecs(*args, **kwargs)
Bases:
Protocol
Ksconf plugin specifications for all known supported functions.
Grouping these functions together in a single class allows for type support it supports typing. This adds a level of validation to the code base where a hook is invoked via
plugin_manger.hook.<hook_name>()
.If you are implementing one of these hooks, please note that you can simple make top-level function, no need to implement a class.
- static ksconf_cli_init()
Simple hook that is run before CLI initialization. This can be use to modify the runtime environment.
This can be used to register additional handlers, such as:
ksconf.combine.register_handler()
- Add a combination file handler. File types are limited to pattern matching.ksconf.layer.register_file_handler()
- Add file handlers for layer processing for template processing
- static ksconf_cli_modify_argparse(parser: Any, name: str)
Manipulate argparse rules. This could be used to add additional CLI options for other hook-added features added features
Note that this hook is called for both the top-level argparse instance as well as each subparser. The
name
argument should be inspected to determine if the parse instances is the parent (top-level) parser, or some other named subcommands.
- static ksconf_cli_process_args(args: Any)
Hook to capture all parsed arguments, includes any custom arguments added to the CLI via the the
ksconf_cli_modify_argparse()
hook.args
can be mutated directly, if needed.
- static modify_jinja_env(env: Any)
Modify the Jinja2 environment object. This can be used to add custom filters or tests, for example.
Invoked by
LayerFile_Jinja2
immediately after initial Environment creation.env
should be mutated in place.
- static package_pre_archive(app_dir: Path, app_name: str)
Modify, inventory, or test the contents of an app before the final packaging commands. This can be triggered from the
ksconf package
command or via the API.During a
ksconf package
process, this hook executes right before the final archive is created. All local merging, app version or build updates, and so on are completed before this hook is executed.From an API perspective, this hook is called from
ksconf.package.AppPackager
whenever a content freeze occurs, which is typically whenmake_archive()
ormake_manifest()
is invoked.
- static post_combine(target: Path, usage: str)
Trigger a custom action after a layer combining operation. This is used by multiple ksconf subcommands and the API.
This trigger could be used to modify the file system, trigger external operations, track/audit behaviors, and so on.
When using CLI commands,
usage
should be either “combine” or “package” depending on which ksconf command was invoked. Direct invocation ofLayerCombiner
can pass along a custom usage label and avoid impacting CLI, when desirable.If your goal is to only trigger an action during the app packaging process, also consider the
package_pre_archive()
hook, which may be more appropriate.
- exception ksconf.hookspec.KsconfPluginWarning
Bases:
Warning
ksconf.layer module
- class ksconf.layer.DirectLayerRoot(context: LayerContext = None)
Bases:
LayerRootBase
A very simple direct LayerRoot implementation that relies on all layer paths to be explicitly given without any automatic detection mechanisms. You can think of this as the legacy implementation.
- add_layer(path: Path)
- order_layers()
- class ksconf.layer.DotDLayerRoot(context=None)
Bases:
LayerRootBase
- class Layer(name: str, root: Path, physical: PurePath, logical: PurePath, context: LayerContext, file_factory: Callable, prune_points: set[Path] = None)
Bases:
Layer
- prune_points: set[Path]
- walk() Iterator[tuple[Path, list[str], list[str]]]
- apply_filter(layer_filter: LayerFilter)
Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
- layer_regex = re.compile('(?P<layer>\\d\\d-[\\w_.-]+)')
- mount_regex = re.compile('(?P<realname>[\\w_.-]+)\\.d$')
- order_layers()
- set_root(root: Path, follow_symlinks=None)
Set a root path, and auto discover all ‘.d’ directories.
Note: We currently only support
.d/<layer>
directories, a file likedefault.d/10-props.conf
won’t be handled here. A valid name would bedefault.d/10-name/props.conf
.
- class ksconf.layer.FileFactory
Bases:
object
- disable(name)
- enable(name, _enabled=True)
- list_available_handlers() list[str]
- register_handler(name: str, **kwargs)
- class ksconf.layer.LayerContext(follow_symlink: 'bool' = False, block_files: 'Match' = re.compile('\\.(bak|swp)$'), block_dirs: 'set' = <factory>, template_variables: 'dict' = <factory>)
Bases:
object
- block_dirs: set
- block_files: Match = re.compile('\\.(bak|swp)$')
- follow_symlink: bool = False
- template_variables: dict
- exception ksconf.layer.LayerException
Bases:
Exception
- class ksconf.layer.LayerFile(layer: Layer, relative_path: PurePath, stat: stat_result = None)
Bases:
PathLike
Abstraction of a file within a Layer
Path definitions
logical_path
Conceptual file path. This is the final path after all layers are resolved. Think of this as the ‘destination’ file.
physical_path
Actual file path. The location of the physical file found within a source layer. Most of the time this is the ‘source’ file, however this doesn’t take into considerations layer combining or template expansion requirements. (In the case of a template, this would be the template file)
resource_path
Content location. Often this the
physical_path
, but in the case of abstracted layers (like templates, or archived layers), this would be the location of a temporary resource that contains the expanded/rendered content.
- layer
- property logical_path: Path
- static match(path: PurePath)
- property mtime
- property physical_path: Path
- relative_path
- property resource_path: Path
- property size
- property stat: stat_result
- class ksconf.layer.LayerFile_Jinja2(*args, **kwargs)
Bases:
LayerRenderedFile
- property jinja2_env
- static match(path: PurePath)
- render(template_path: Path) str
- static transform_name(path: PurePath)
- class ksconf.layer.LayerRenderedFile(*args, **kwargs)
Bases:
LayerFile
Abstract LayerFile for rendered scenarios, such as template scenarios. A subclass really only needs to implement
match()
render()
- property logical_path: Path
- property physical_path: Path
- render(template_path: Path) str
- property resource_path: Path
- static transform_name(path: PurePath)
- use_secure_delete = False
- class ksconf.layer.LayerRootBase(context: LayerContext = None)
Bases:
object
All ‘path’s here are relative to the ROOT.
- class Layer(name: str, root: Path, physical: PurePath, logical: PurePath, context: LayerContext, file_factory: Callable)
Bases:
object
Basic layer Container: Connects logical and physical paths.
- context
- logical_path
- name
- physical_path
- root
- walk() Iterator[tuple[Path, list[str], list[str]]]
- apply_filter(layer_filter: LayerFilter) bool
Apply a destructive filter to all layers.
layer_filter(layer)
will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.Returns True if layers were removed
- list_layer_names() list[str]
- order_layers()
- exception ksconf.layer.LayerUsageException
Bases:
LayerException
- ksconf.layer.register_file_handler(name: str, **kwargs)
ksconf.package module
- class ksconf.package.AppPackager(src_path, app_name: str, output: TextIO, template_variables: dict = None, predictable_mtime: bool = True)
Bases:
object
- block_local(report=True)
- blocklist(patterns)
- check()
Run safety checks prior to building archive:
Set app name based on app.conf [package] id, if set. Otherwise confirm that the package id and top-level folder names align.
Check for files or directories starting with
.
, makes AppInspect very grumpy!
- cleanup()
- combine(src, filters, layer_method='dir.d', allow_symlink=False)
- expand_new_only(value: str) str | None
Expand a variable but return False if no substitution occurred
- Parameters:
value (str) – String that may contain
{{variable}}
substitution.- Returns:
Expanded value if variables were expanded, else False
- Return type:
str
- expand_var(value: str) str
Expand a variable, if present
- Parameters:
value (str) – String that main contain
{{variable}}
substitution.- Returns:
Expanded value
- Return type:
str
- freeze(caller_name)
Initiate a content freeze by restricting mutable methods. The “package_pre_archive” hook is invoked before freeze operation. Such hooks may choose to mutate the filesystem at
app_dir
, the only assumption is that all work is done before the hook returns.Freeze can be safely called multiple times.
caller_name
is simply a label used in an exception message if the programmer screwed up.
- make_archive(filename, temp_suffix: str = '.tmp')
Create a compressed tarball of the build directory.
- make_manifest(calculate_hash=True) AppManifest
Create a manifest of the app’s contents.
- merge_local()
Find everything in local, if it has a corresponding file in default, merge.
- require_active_context(mutable=True)
Decorator to mark member functions that cannot be used until the context manager has been activated.
- update_app_conf(version: str = None, build: str = None)
Update version and/or build in
apps.conf
- class ksconf.package.AppVarMagic(src_dir, build_dir, meta=None)
Bases:
object
A lazy loading dict-like object to fetch things like app version and such on demand.
- expand(value: str) str
A simple Jinja2 like
{{VAR}}
substitution mechanism.
- get_app_id()
Splunk app package id from app.conf
- get_build()
Splunk app build fetched from app.conf
- get_git_head()
Git HEAD rev abbreviated
- get_git_last_rev()
Git abbreviated rev of the last change of the app. This may not be the same as HEAD.
- get_git_tag()
Git version tag using the
git describe --tags
command
- get_layers_hash()
Build a unique hash representing the combination of ksconf layers used.
- get_layers_list()
List of ksconf layers used.
- get_version()
Splunk app version fetched from app.conf
- git_single_line(*args)
- list_vars()
Return a list of (variable, description) available in this class.
- exception ksconf.package.AppVarMagicException
Bases:
KeyError
- exception ksconf.package.PackagingException
Bases:
Exception
- ksconf.package.find_conf_in_layers(app_dir, conf, *layers)
- ksconf.package.get_merged_conf(app_dir, conf, *layers)
- ksconf.package.normalize_directory_mtime(path)
Walk a tree and update the directory modification times to match the newest time of the children. This results in a more predictable behavior over multiple executions.
ksconf.setup_entrypoints module
Defines all command prompt entry points for CLI actions
- This is a silly hack allows for fallback mechanism when
running unit tests (can happen before install)
unexpected issues with importlib.metadata or backport
- class ksconf.setup_entrypoints.Ep(name, module_name, object_name)
Bases:
NamedTuple
- property formatted
- module_name: str
Alias for field number 1
- name: str
Alias for field number 0
- object_name: str
Alias for field number 2
- class ksconf.setup_entrypoints.LocalEntryPoint(data)
Bases:
object
Bare minimum stand-in for entrypoints.EntryPoint
- load()
- ksconf.setup_entrypoints.debug()
- ksconf.setup_entrypoints.get_entrypoints_fallback(group)
- ksconf.setup_entrypoints.get_entrypoints_setup()
Build entry point text descriptions for ksconf packaging
ksconf.xmlformat module
- class ksconf.xmlformat.FileReadlinesCache
Bases:
object
Silly workaround for CDATA detection…
- static convert_filename(filename)
- readlines(filename)
- class ksconf.xmlformat.SplunkSimpleXmlFormatter
Bases:
object
- static cdata_tags(elem: Any, tags: List[str])
Expand text to CDATA, if it isn’t already.
- classmethod expand_tags(elem: Any, tags: set)
Keep <elem></elem> instead of shortening to <elem/>
- classmethod format_json(elem: Any, indent=2)
Format JSON data within a Dashboard Studio dashboard. This is still pretty limited (for example, long searches still show up on a single line), but this give you at least a fighting change to figure out what’s different.
- classmethod format_xml(src, dest, default_indent=2)
- static guess_indent(elem: Any, default=2)
- classmethod indent_tree(elem: Any, level=0, indent=2)
- keep_tags = {'default', 'earliest', 'fieldset', 'label', 'latest', 'option', 'search', 'set'}