My major init_app_conf module

Description of high-level API module at AlexBerUtils project

ymlparsers and parser modules serves as Low-Level API for this module. Here is the story about ymlparsers and here is is the story about parser.

The source code you can found here. It is available as part of my AlexBerUtils s project.

You can install AlexBerUtils from PyPi:

python3 -m pip install -U alex-ber-utils

You should also install some optional dependencies, such as jinja2. The easiest way is to run:

python3 -m pip install alex-ber-utils[yml]

Note also that hiyapyco should be at least 0.4.16.

See here for more details explanation on how to install.

Code example

Put the following files to the same directory as the script above.

Note: names of the configuration files are important.

config.yml:

config-dev.yml

config-local.yml

Now, I’m going to give you an overview on what’s going on and below I will give your more details explanation.

Let’s start from configuration files. Config.yml is the default configuration file name (it can be changed, see below). It has general.profiles section that contains additional “profiles” to be applied. Practically, it means, that if we have [‘dev’, ‘local’] there, it means that in addition to this file config-dev.yml and config-local.yml will be also loaded, and applied in that order. You can think about it as applying them one after another, and if there is some attribute that was defined in previous file, it is just replaced. One example, of profile is your execution environment. You may want to change some of you configuration parameter, depending if you’re running your script locally, in your development setup (dev) or in production (prod).

Here I’m changing app.host_name — it will be bing.com (locally), yahoo.com (dev) and google.com (default and prod). Also there is different log.root.level and log.root.handlers.

Default and prod log.root.level is DEBUG, dev’s log.root.level is INFO and locall’s log.root.level is DEBUG (again).

Default and prod log.root.handlersis [all_file_handler], dev’s log.root.handlersis unchanged, it remains the same, and locall’s log.root.handlers is [console]. This means that locally we will have log output only in the console, and in other environment both in the console (stdout) and in the file logs/app.log. Note: You should create logs directory in all but locall environment before you start the script, typically, it will be soft link to the machine logs directory. Local environment will not redirect logs to the file.

app.news in all files has the same value, so you will see it any every environment (under the hood it will picked up from different files; it is done here for demonstration purposes).

All other attribute will have value as described in the default config.yml.

You can see that configuration has 2 parts: generaland app. general part is, generally speaking, the part that has attribute with “special meaning” such general.profiles.You can put also there also some unrelated stuff, such as log.You will see in a bit why I think it is ok.

Most of the attribute should be put in app section. These attributes that are specific to your application.

Note: You can also override attribute through system argument. You will see in a bit.

Let’s go over the code snippet.

Lines 1–5 are imports.

On line 7, I’m created global logger attribute and initialize it to None.

Lines 10–12, I’m extending library class alexber.utils.init_app_conf.conf It is used as attribute names. It is just convenient replacement for the string it holds. alexber.utils.init_app_conf.conf has attributes with special meanings.

Line 17 has definition of run() function that will be called after all configuration will be parsed. It will receive them as kwargs.

Line 21 has definition of _log_config() function that receives parsed configuration, pops log’s configuration and initialize log.

Line 31 has definition of main() function that receives optional args parameter. It has default None value. If no explicit value is passed than sys.args will be used implicitly. This function can be called from another module.

Lines 50–51 — standard code snippet: if this module executes as __main__ (and not imported to another module), than after all methods and (global) module-level attributes will be defined we’re calling main() function.

When this code is executed, after all methods and (global) module-level attributes are define, main() function is called with args=None.

On lines 33–34 I’m making temporary logger initialization, when all logs with level INFO and above are redirected to stderr, all warnings are redirected to stderr. This will be effective till we’ll parse log’s configuration and reinitialize logger. This is done, in order to see log’s output, especially warnings. See Integrating Python’s logging and warnings packages for more details.

On line 37 we makes relative path to file to work.

Lines 39–42 responsible for parsing configuration files. I will get back to them in a while.

On line 42 we receives dict that has parsed configuration.

On line 44 we’re passing it to _log_config() function. Their we’re poping general.log part from the dict config and we’re calling logging.config.dictConfig() to reinitialize logger.

On lines 23–24 we’re initialize global logger attribute of current module, based on provided configuration.

On lines 26–27 we’re pretty-print parsed configuration. Uncommented line uses ymlparsers.as_str() as convenient method to print logger configuration to the logger. Commented-out line us pretty-print pprint module (it is available in standard Python). It has caveat, though.

Note: Actual type is OrderedDict and not dict, but it is mainly for historical reasons…

Side note: Up to (not included) Python 3.6 the order in which key/value are stored was undefined. In Python 3.6 it was stated that this is implementation detail of CPython (and best practice is not to relay on this behavior). Started from Python 3.7 dictionary order is guaranteed to be insertion order.

See https://stackoverflow.com/a/58676384/1137529 for the differences between dict and OrderedDict.

https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

pprint was wriiten way before 3.6. In order to produce consistent result, it sorts out dict by key. I have found this unappropriated, I want to see the configuration in exact same order as it defined in the configuration file. In Python 3.8, however, you can specify sort_dicts=False in order to disable this sorting functionality. So, if you’re using Python 3.8 or above this will also works.

In line 45 we just remove general.log from parsed result. You can change remove this line, if you want to keep log configuration.

In line 46 we’re calling run() function where actual business logic should be put. In this example, it just logs run().

I’ve skipped details description of init_app_conf.parse_config() function. I will start from init_app_conf.initConig() and I then I’ll describe init_app_conf.parse_config().

initConig() function can be optionally called prior any call to another function in this module. In the code snippet above I didn’t have such call. It is indented to be called in the MainThread. This function can be call with empty params. This function is idempotent. Parameters are :

• default_parser_cls can be class or str. Optional. Default values is: AppConfParser This enables you to customize the parsing configuration logic for your needs.
• default_parser_kwargs this params will be used as default values in default_parser_cls.__init__() function. Default values are ‘implicit_convert’:True, This means, by default converting config values is done using mask_value() function (see below).

parse_config() function. This is the main function of the module. I remind you, that ymlparsers and parser modules serves as Low-Level API for this module. So, if you want to change the default behavior, you should call ymlparsers.intConfig() first. See My ymlparsers module for more details.

What does this function do?

• This function parses command line arguments first.
• Than it parse yml files.
• Command line arguments overrides yml files arguments.

Parameters of yml files we always try to convert on best-effort basses. Parameters of system args we try convert according to implicit_convert param.

In more detail, command line arguments of the form --key=value are parsed first. If exists --config_file it’s value is used to search for yml file. if --config_file is absent, ‘config.yml’ is used for yml file. If yml file is not found, only command line arguments are used. If yml file is found, both arguments are used, while command line arguments overrides yml arguments.

--general.profiles or appropriate key in default yml file is used to find ‘profiles’. Let suppose, that --config_file is resolved to config.yml. If ‘profiles’ is not empty, than it will be used to calculate filenames that will be used to override default yml file. Let suppose, ‘profiles’ resolved to [‘dev’, ‘local’] as in our code snippet. Than first ‘config.yml’ will be loaded, than it will be overridden with config-dev.yml, than it will be overridden with config-local.yml. At last, it will be overridden with system args. This entry can be always be overridden with system args.

general.whiteListSysOverride key in yml file is optional. If not provided, than any key that exists in the default yml file can be overridden with system args. If provided, than only key that start with one of the key provided here can be used to override entrys with system args. This entry can’t be overridden with system args.

--general.listEnsure or appropriate key in default yml file is used to instruct that listed key should be interpreted as comma-delimited list when is used to override entrys with system args. This entry can be always be overridden with system args.

general.config.file is key that is used in returned dict that points to default yml file.

If implicit_convert=True, than for system args we assume Python built-in types, including bool, and we're converting it to appropriate type.
Otherwise, implicit_convert will have the value that was set in initConig(). By default it is True. See mask_value() function below.

Parameters:

• argumentParser. Can be None. You can instantiate ArgumentParser, for example, with default values. See here for example, or read documentation of ArgumentParser here. Of course, you can just pass None and make post-process for the populated dict.
• args: if not None, suppresses sys.args.
• implicit_convert: if None, than value that was passed to initConig() is used (default). if True value attempt to convert value from system args to appropriate type will be done, if False value from system args will be used as is. See mask_value() function below.

mask_value() function implemented as a wrapper to parsers.safe_eval() method with support for boolean variables. Boolvalues are case-insensitive.

This function is used inside init_app_conf to get type for arguments that we get from system args.

This mechanism can be easily replaced with your own one, just provide default_parser_cls, in initConig() function.

to_convex_map() utility function receives dictionary with flat keys, it has simple key:value structure where value can’t be another dictionary. This method is not used in AlexBerUtils project. It will return dictionary of dictionaries with natural key mapping (see bellow), optionally, entries will be filtered out according to white_list_flat_keys and, optionally, value will be implicitly converted to appropriate type.

In order to simulate dictionary of dictionaries flat keys compose key from outer dict with key from inner dict separated with dot. For example, general.profiles flat key corresponds to convex map with general key with dictionary as value that have one of the keys profiles with corresponding value.

If white_list_flat_keys is not None, it will be used to filter out entries from the dict with flat keys.

If implicit_convert=True, than for system args we assume Python built-in types, including bool, and we're converting it to appropriate type.
Otherwise, implicit_convert will have the value that was set in initConig(). By default it is True. See mask_value() function above.

Parameters:

• d: dict with flat keys.
• white_list_flat_keys:Optional. if present, only keys that start with one of the elements listed here will be considered.
• implicit_convert: if None, than value that was passed to initConig() is used (default). if True value attempt to convert value from system args to appropriate type will be done, if False value from system args will be used as is. See mask_value() function above.

merge_list_value_in_dicts() utility function merges value of 2 dicts. This value represents list of values. This function returns merged converted value, typically one from flat_d, if it is empty than from d.This function is used in deploys module (see link below).

Parameters:

• flat_d: flat dictionary, usually one that was created from parsing system args.
• d: dictionary of dictionaries, usually one that was created from parsing YAML file.
• main_key: d[main_key] is absent or dict. See below.
• sub_key: d[main_key][sub_key]is absent or list. See below.
• implicit_convert: if None, than value that was passed to initConig() is used (default). if True value attempt to convert value from system args to appropriate type will be done, if False value from system args will be used as is. See mask_value() function above.

Value from flat_d is roughly obtained by flat_d[main_key+’.’+sub_key].

Value from d is roughly obtained by d[main_key][sub_key].

If value (or intermediate value) is not found empty dict is used.

This method assumes that flat_d value contain str that represent list (comma-delimited).

This method assumes that d[main_key] contains dict. implicit_convert is applied only for flat_d.

If implicit_convert=True, than for system args we assume Python built-in types, including bool, and we're converting it to appropriate type.
Otherwise, implicit_convert will have the value that was set in initConig(). By default it is True. See mask_value() function above.

This function returns merged converted value, typically one from flat_d, if it is empty than from d.

See also:

• importer module or How to write easily customizable code?
• fixabscwd() function in mains module or Making relative path to file to work.
• fix_retry_env() function in mains module or Make path to file on Windows works on Linux.
• parser module or Description of one the oldest AlexBerUtils project
• ymlparsers module or Description of low-level API module for another modules
• init_app_conf module, or My major init_app_con module
• deploys module, or My deploys module
• emails module,or My emails module
• processinvokes module, or My processinvokes module
• stdLogging module, or My stdLogging Module