ksconf package¶
Subpackages¶
- ksconf.app package
- ksconf.builder package
- ksconf.commands package
- Submodules
- ksconf.commands.check module
- ksconf.commands.combine module
- ksconf.commands.diff module
- ksconf.commands.filter module
- ksconf.commands.merge module
- ksconf.commands.minimize module
- ksconf.commands.package module
- ksconf.commands.promote module
- ksconf.commands.restexport module
- ksconf.commands.restpublish module
- ksconf.commands.snapshot module
- ksconf.commands.sort module
- ksconf.commands.unarchive module
- ksconf.commands.xmlformat module
- Module contents
- ksconf.conf package
- ksconf.util package
- ksconf.vc package
Submodules¶
ksconf.archive module¶
-
class
ksconf.archive.
GenArchFile
(path, mode, size, payload)¶ Bases:
tuple
-
mode
¶ Alias for field number 1
-
path
¶ Alias for field number 0
-
payload
¶ Alias for field number 3
-
size
¶ Alias for field number 2
-
-
ksconf.archive.
extract_archive
(archive_name, extract_filter: callable = None) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
gaf_filter_name_like
(pattern)¶
-
ksconf.archive.
gen_arch_file_remapper
(iterable: Iterable[ksconf.archive.GenArchFile], mapping: Sequence[Tuple[str, str]]) → Iterable[ksconf.archive.GenArchFile]¶
-
ksconf.archive.
sanity_checker
(iterable: Iterable[ksconf.archive.GenArchFile]) → Iterable[ksconf.archive.GenArchFile]¶
ksconf.cli module¶
KSCONF - Ksconf Splunk CONFig tool
Optionally supports argcomplete for commandline argument (tab) completion.
Install & register with:
pip install argcomplete activate-global-python-argcomplete (in ~/.bashrc)
-
ksconf.cli.
build_cli_parser
(do_formatter=False)¶
-
ksconf.cli.
check_py
()¶
-
ksconf.cli.
check_py_sane
()¶ Run a simple python environment sanity check. Here’s the scenario, if Splunk’s python is called but not all the correct environment variables have been set, then ksconf can fail in unclear ways.
-
ksconf.cli.
cli
(argv=None, _unittest=False)¶
-
ksconf.cli.
handle_cmd_failed
(subparser, ep)¶ Build a bogus subparser for a cmd that can’t be loaded, with the only purpose of providing a more consistent user experience.
ksconf.combine module¶
XXX: Move the overwrite-as-necessary logic into a subclass; for several use cases we just don’t care because ‘target’ is a brand new directory
-
class
ksconf.combine.
LayerCombiner
(follow_symlink: bool = False, banner: str = '', dry_run: bool = False, quiet: bool = False)¶ Bases:
object
Class to recursively combine layers (directories) into a single rendered output target directory. This is heavily used by the
ksconf combine
command as well as by thepackage
command.Typical class use case:
- ::
lc = LayerCombiner()
- # Setup source, either
- lc.set_source_dirs() OR
- lc.set_layer_root()
Call hierarch:
lc.combine() Entry point -> prepare() Directory, layer prep -> prepare_target_dir() Make dir; subclass handles marker here (combine CLI) -> pre_combine_inventory() Hook for pre-processing (or alerting) the set of files to combine -> combine_files() Main worker function -> post_combine() Optional, cleanup leftover files
-
add_layer_filter
(action, pattern)¶
-
combine
(target)¶ Combine layers into
target
directory.
-
combine_files
(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile])¶
-
conf_file_re
= re.compile('([a-z_-]+\\.conf|(default|local)\\.meta)$')¶
-
filetype_handlers
= [(<function LayerCombiner.register_regex.<locals>.match_f>, <function handle_merge_conf_files>), (<function LayerCombiner.register_regex.<locals>.match_f>, <function handle_spec_concatenate>)]¶
-
post_combine
(target)¶ Hook point for post-processing after all copy/merge operations have been completed.
-
pre_combine_inventory
(target: pathlib.Path, src_files: List[ksconf.layer.LayerFile]) → List[ksconf.layer.LayerFile]¶ Hook point for pre-processing before any files are copied/merged
-
prepare
(target: pathlib.Path)¶ Start the combine process. This includes directory checking, applying layer filtering, and marker file handling.
-
prepare_target_dir
(target: pathlib.Path)¶ Hook to ensure destination directory is ready for use. This can be overridden to adder marker file handling for use cases that need it (e.g., the ‘combine’ command)
-
classmethod
register_regex
(regex_match)¶ Decorator that matches a filename regex and if it matches, it executes the decorator handler.
-
set_layer_root
(root: ksconf.layer.LayerRootBase.Layer)¶
-
set_source_dirs
(sources: List[pathlib.Path])¶
-
spec_file_re
= re.compile('\\.conf\\.spec$')¶
-
exception
ksconf.combine.
LayerCombinerException
¶ Bases:
Exception
-
ksconf.combine.
handle_merge_conf_files
(context: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)¶
-
ksconf.combine.
handle_spec_concatenate
(context: ksconf.combine.LayerCombiner, dest_path: pathlib.Path, sources: List[ksconf.layer.LayerFile], dry_run)¶
ksconf.compat module¶
Silly simple Python version compatibility items
-
ksconf.compat.
cache
(user_function)¶
-
ksconf.compat.
handle_py3_kw_only_args
(kw, *default_args)¶ Fake support for Python 3.8+ style keyword-only style arguments, or
*
arg syntax.Example Python 3.8+ syntax:
def f(arg, *args, a=True, b=False): ...
Example Python 3.7 (and earlier) syntax with this helper function:
def f(arg, *args, **kw_only): a, b = handle_py3_kw_only_args(kw_only, ("a", True), ("b", False)) ...
Parameters: - kw (dict) – keyword arguments provided to the calling function. Be aware that this dict will be empty after this function is done.
- default_args (tuple) – pairs of keyword argument to the caller function in argument (arg_name, default_value) order.
Raises: TypeError – if
kw
contains any keys not defined inargs
This mirrors Python’s native behavior when an unexpected argument is passed to a function.
ksconf.consts module¶
ksconf.filter module¶
-
class
ksconf.filter.
FilteredList
(flags=0, default=True)¶ Bases:
object
-
IGNORECASE
= 1¶
-
INVERT
= 2¶
-
VERBOSE
= 4¶
-
feed
(item, filter=None)¶
-
feedall
(iterable, filter=None)¶
-
has_rules
¶
-
match
(item)¶
-
match_path
(path)¶
-
match_stanza
(stanza)¶ Same as match(), but handle GLOBAL_STANZA gracefully.
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListRegex
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Regular Expression support
-
calc_regex_flags
()¶
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListSplunkGlob
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Classic wildcard support (‘*’ and ?’) plus ‘…’ or ‘**’ for multiple-path components with some (non-advertised) pass-through regex behavior
-
class
ksconf.filter.
FilteredListString
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredList
Handle simple string comparisons
-
reset_counters
()¶
-
-
class
ksconf.filter.
FilteredListWildcard
(flags=0, default=True)¶ Bases:
ksconf.filter.FilteredListRegex
Wildcard support (handling ‘*’ and ?’) Technically fnmatch also supports [] and [!] character ranges, but we don’t advertise that
-
ksconf.filter.
create_filtered_list
(match_mode, flags=0, default=True)¶
ksconf.layer module¶
-
class
ksconf.layer.
DirectLayerRoot
(config: Optional[ksconf.layer.LayerConfig] = None)¶ Bases:
ksconf.layer.LayerRootBase
A very simple direct LayerRoot implementation that relies on all layer paths to be explicitly given without any automatic detection mechanisms. You can think of this as the legacy implementation.
-
add_layer
(path: pathlib.Path)¶
-
order_layers
()¶
-
-
class
ksconf.layer.
DotDLayerRoot
(config=None)¶ Bases:
ksconf.layer.LayerRootBase
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, config: ksconf.layer.LayerConfig, file_factory: Callable, prune_points: Optional[Set[pathlib.Path]] = None)¶ Bases:
ksconf.layer.Layer
-
prune_points
¶
-
walk
() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]¶
-
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter)¶ Apply a destructive filter to all layers. layer_filter(layer) will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.
Returns True if layers were removed
-
layer_regex
= re.compile('(?P<layer>\\d\\d-[\\w_.-]+)')¶
-
list_layers
() → List[Layer]¶
-
mount_regex
= re.compile('(?P<realname>[\\w_.-]+)\\.d$')¶
-
order_layers
()¶
-
set_root
(root: pathlib.Path, follow_symlinks=None)¶ Set a root path, and auto discover all ‘.d’ directories.
Note: We currently only support ‘.d/<layer>’ directories, a file like default.d/10-props.conf won’t be handled here.
-
class
-
class
ksconf.layer.
LayerConfig
¶ Bases:
object
-
exception
ksconf.layer.
LayerException
¶ Bases:
Exception
-
class
ksconf.layer.
LayerFile
(layer: ksconf.layer.LayerRootBase.Layer, relative_path: pathlib.PurePath, stat: Optional[os.stat_result] = None)¶ Bases:
os.PathLike
-
layer
¶
-
logical_path
¶
-
static
match
(path: pathlib.PurePath)¶
-
mtime
¶
-
physical_path
¶
-
relative_path
¶
-
resource_path
¶
-
size
¶
-
stat
¶
-
-
class
ksconf.layer.
LayerFilter
¶ Bases:
object
-
add_rule
(action, pattern)¶
-
evaluate
(layer: ksconf.layer.LayerRootBase.Layer) → bool¶
-
-
class
ksconf.layer.
LayerRootBase
(config: Optional[ksconf.layer.LayerConfig] = None)¶ Bases:
object
All ‘path’s here are relative to the ROOT.
-
class
Layer
(name: str, root: pathlib.Path, physical: pathlib.PurePath, logical: pathlib.PurePath, config: ksconf.layer.LayerConfig, file_factory: Callable)¶ Bases:
object
Basic layer Container: Connects logical and physical paths.
-
config
¶
-
get_file
(path: pathlib.Path) → ksconf.layer.LayerFile¶ Return file object (by logical path), if it exists in this layer.
-
iter_files
() → Iterator[ksconf.layer.LayerFile]¶
-
list_files
() → List[ksconf.layer.LayerFile]¶
-
logical_path
¶
-
name
¶
-
physical_path
¶
-
root
¶
-
walk
() → Iterator[Tuple[pathlib.Path, List[str], List[str]]]¶
-
-
add_layer
(layer: Layer, do_sort=True)¶
-
apply_filter
(layer_filter: ksconf.layer.LayerFilter) → bool¶ Apply a destructive filter to all layers. layer_filter(layer) will be called one for each layer, if the filter returns True than the layer is kept. Root layers are always kept.
Returns True if layers were removed
-
get_file
(path) → Iterator[ksconf.layer.LayerFile]¶ return all layers associated with the given relative path.
-
get_layers_by_name
(name: str) → Iterator[ksconf.layer.LayerRootBase.Layer]¶
-
iter_all_files
() → Iterator[ksconf.layer.LayerFile]¶ Iterator over all physical files.
-
list_files
() → List[ksconf.layer.LayerFile]¶ Return a list of logical paths.
-
list_layer_names
() → List[str]¶
-
list_layers
() → List[Layer]¶
-
list_logical_files
() → List[ksconf.layer.LayerFile]¶ Return a list of logical paths.
-
list_physical_files
() → List[ksconf.layer.LayerFile]¶
-
order_layers
()¶
-
class
-
exception
ksconf.layer.
LayerUsageException
¶ Bases:
ksconf.layer.LayerException
-
class
ksconf.layer.
TemplatedLayerFile
(*args, **kwargs)¶ Bases:
ksconf.layer.LayerFile
-
logical_path
¶
-
static
match
(path: pathlib.PurePath)¶
-
physical_path
¶
-
render
(template_path: pathlib.Path) → str¶
-
resource_path
¶
-
template_context
= {}¶
-
static
transform_name
(path: pathlib.PurePath)¶
-
-
ksconf.layer.
layer_file_factory
(layer, path: pathlib.PurePath, *args, **kwargs) → ksconf.layer.LayerFile¶
-
ksconf.layer.
path_in_layer
(layer: pathlib.Path, path: pathlib.Path) → pathlib.Path¶ Check to see if path exist within layer. Returns either None, or the path without the shared prefix with layer.
ksconf.package module¶
-
class
ksconf.package.
AppPackager
(src_path, app_name: str, output: TextIO)¶ Bases:
object
-
block_local
(report=True)¶
-
blocklist
(patterns)¶
-
check
()¶ Run safety checks prior to building archive:
- Set app name based on app.conf [package] id, if set. Otherwise confirm that the package id and top-level folder names align.
- Check for files or directories starting with
.
, makes AppInspect very grumpy!
-
cleanup
()¶
-
combine
(src, filters, layer_method='dir.d', allow_symlink=False)¶
-
expand_new_only
(value: str) → Optional[str]¶ Expand a variable but return False if no substitution occurred
Parameters: value (str) – String that may contain {{variable}}
substitution.Returns: Expanded value if variables were expanded, else False Return type: str
-
expand_var
(value: str) → str¶ Expand a variable, if present
Parameters: value (str) – String that main contain {{variable}}
substitution.Returns: Expanded value Return type: str
-
make_archive
(filename)¶ Create a compressed tarball of the build directory.
-
make_manifest
(calculate_hash=True) → ksconf.app.manifest.AppManifest¶ Create a manifest of the app’s contents.
-
merge_local
()¶ Find everything in local, if it has a corresponding file in default, merge.
-
require_active_context
()¶ Decorator to mark member functions that cannot be used until the context manager has been activated.
-
update_app_conf
(version: str = None, build: str = None)¶ Update version and/or build in
apps.conf
-
-
class
ksconf.package.
AppVarMagic
(src_dir, build_dir, meta=None)¶ Bases:
object
A lazy loading dict-like object to fetch things like app version and such on demand.
-
expand
(value: str) → str¶ A simple Jinja2 like {{VAR}} substitution mechanism.
-
get_app_id
()¶ Splunk app package id from app.conf
-
get_build
()¶ Splunk app build fetched from app.conf
-
get_git_head
()¶ Git HEAD rev abbreviated
-
get_git_last_rev
()¶ Git abbreviated rev of the last change of the app. This may not be the same as HEAD.
-
get_git_tag
()¶ Git version tag using the
git describe --tags
command
-
get_layers_hash
()¶ Build a unique hash representing the combination of ksconf layers used.
-
get_layers_list
()¶ List of ksconf layers used.
-
get_version
()¶ Splunk app version fetched from app.conf
-
git_single_line
(*args)¶
-
list_vars
()¶ Return a list of (variable, description) available in this class.
-
-
exception
ksconf.package.
AppVarMagicException
¶ Bases:
KeyError
-
exception
ksconf.package.
PackagingException
¶ Bases:
Exception
-
ksconf.package.
find_conf_in_layers
(app_dir, conf, *layers)¶
-
ksconf.package.
get_merged_conf
(app_dir, conf, *layers)¶
-
ksconf.package.
normalize_directory_mtime
(path)¶ Walk a tree and update the directory modification times to match the newest time of the children. This results in a more predictable behavior over multiple executions.
ksconf.setup_entrypoints module¶
Defines all command prompt entry points for CLI actions
- This is a silly hack allows for fallback mechanism when
- running unit tests (can happen before install)
- if entrypoints or pkg_resources are not available at run time (Splunk’s embedded python)
-
class
ksconf.setup_entrypoints.
Ep
(name, module_name, object_name)¶ Bases:
tuple
-
module_name
¶ Alias for field number 1
-
name
¶ Alias for field number 0
-
object_name
¶ Alias for field number 2
-
-
class
ksconf.setup_entrypoints.
LocalEntryPoint
(data)¶ Bases:
object
Bare minimum stand-in for entrypoints.EntryPoint
-
load
()¶
-
-
ksconf.setup_entrypoints.
debug
()¶
-
ksconf.setup_entrypoints.
get_entrypoints_fallback
(group)¶
-
ksconf.setup_entrypoints.
get_entrypoints_setup
()¶
ksconf.xmlformat module¶
-
class
ksconf.xmlformat.
FileReadlinesCache
¶ Bases:
object
Silly workaround for CDATA detection…
-
static
convert_filename
(filename)¶
-
readlines
(filename)¶
-
static
-
class
ksconf.xmlformat.
SplunkSimpleXmlFormatter
¶ Bases:
object
Expand text to CDATA, if it isn’t already.
Keep <elem></elem> instead of shortening to <elem/>
-
classmethod
format_json
(elem: Any, indent=2)¶ Format JSON data within a Dashboard Studio dashboard. This is still pretty limited (for example, long searches still show up on a single line), but this give you at least a fighting change to figure out what’s different.
-
classmethod
format_xml
(src, dest, default_indent=2)¶
-
static
guess_indent
(elem: Any, default=2)¶
-
classmethod
indent_tree
(elem: Any, level=0, indent=2)¶
Module contents¶
ksconf - Ksconf Splunk CONFig tool
Design goals:
- Multi-purpose go-to
.conf
tool.- Dependability
- Simplicity
- No eternal dependencies (single source file, if possible; or packable as single file.)
- Stable CLI
- Good scripting interface for deployment scripts and/or git hooks
-
exception
ksconf.
KsconfPluginWarning
¶ Bases:
Warning