My major init_app_conf module

Description of high-level API module at AlexBerUtils project

ymlparsers and parser modules serves as Low-Level API for this module. Here is the story about ymlparsers and here is is the story about parser.

The source code you can found here. It is available as part of my AlexBerUtils s project.

You can install AlexBerUtils from PyPi:

You should also install some optional dependencies, such as jinja2. The easiest way is to run:

Note also that hiyapyco should be at least 0.4.16.

See here for more details explanation on how to install.

Code example

Put the following files to the same directory as the script above.

Note: names of the configuration files are important.

config.yml:

config-dev.yml

config-local.yml

Now, I’m going to give you an overview on what’s going on and below I will give your more details explanation.

Let’s start from configuration files. Config.yml is the default configuration file name (it can be changed, see below). It has section that contains additional “profiles” to be applied. Practically, it means, that if we have there, it means that in addition to this file and will be also loaded, and applied in that order. You can think about it as applying them one after another, and if there is some attribute that was defined in previous file, it is just replaced. One example, of profile is your execution environment. You may want to change some of you configuration parameter, depending if you’re running your script locally, in your development setup (dev) or in production (prod).

Here I’m changing — it will be bing.com (locally), yahoo.com (dev) and google.com (default and prod). Also there is different and .

Default and prod is DEBUG, dev’s is INFO and locall’s is DEBUG (again).

Default and prod is [all_file_handler], dev’s is unchanged, it remains the same, and locall’s is [console]. This means that locally we will have log output only in the console, and in other environment both in the console () and in the file Note: You should create directory in all but locall environment before you start the script, typically, it will be soft link to the machine logs directory. Local environment will not redirect logs to the file.

in all files has the same value, so you will see it any every environment (under the hood it will picked up from different files; it is done here for demonstration purposes).

All other attribute will have value as described in the default config.yml.

You can see that configuration has 2 parts: and . part is, generally speaking, the part that has attribute with “special meaning” such You can put also there also some unrelated stuff, such as You will see in a bit why I think it is ok.

Most of the attribute should be put in section. These attributes that are specific to your application.

Note: You can also override attribute through system argument. You will see in a bit.

Let’s go over the code snippet.

Lines 1–5 are imports.

On line 7, I’m created global attribute and initialize it to

Lines 10–12, I’m extending library class It is used as attribute names. It is just convenient replacement for the string it holds. has attributes with special meanings.

Line 17 has definition of run() function that will be called after all configuration will be parsed. It will receive them as kwargs.

Line 21 has definition of _log_config() function that receives parsed configuration, pops log’s configuration and initialize log.

Line 31 has definition of main() function that receives optional args parameter. It has default None value. If no explicit value is passed than sys.args will be used implicitly. This function can be called from another module.

Lines 50–51 — standard code snippet: if this module executes as __main__ (and not imported to another module), than after all methods and (global) module-level attributes will be defined we’re calling main() function.

When this code is executed, after all methods and (global) module-level attributes are define, main() function is called with args=None.

On lines 33–34 I’m making temporary logger initialization, when all logs with level INFO and above are redirected to stderr, all warnings are redirected to stderr. This will be effective till we’ll parse log’s configuration and reinitialize logger. This is done, in order to see log’s output, especially warnings. See Integrating Python’s logging and warnings packages for more details.

On line 37 we makes relative path to file to work.

Lines 39–42 responsible for parsing configuration files. I will get back to them in a while.

On line 42 we receives dict that has parsed configuration.

On line 44 we’re passing it to _log_config() function. Their we’re poping general.log part from the dict config and we’re calling logging.config.dictConfig() to reinitialize logger.

On lines 23–24 we’re initialize global logger attribute of current module, based on provided configuration.

On lines 26–27 we’re pretty-print parsed configuration. Uncommented line uses ymlparsers.as_str() as convenient method to print logger configuration to the logger. Commented-out line us pretty-print module (it is available in standard Python). It has caveat, though.

Note: Actual type is and not but it is mainly for historical reasons…

Side note: Up to (not included) Python 3.6 the order in which key/value are stored was undefined. In Python 3.6 it was stated that this is implementation detail of CPython (and best practice is not to relay on this behavior). Started from Python 3.7 dictionary order is guaranteed to be insertion order.

See https://stackoverflow.com/a/58676384/1137529 for the differences between and .

https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

was wriiten way before 3.6. In order to produce consistent result, it sorts out dict by key. I have found this unappropriated, I want to see the configuration in exact same order as it defined in the configuration file. In Python 3.8, however, you can specify in order to disable this sorting functionality. So, if you’re using Python 3.8 or above this will also works.

In line 45 we just remove general.log from parsed result. You can change remove this line, if you want to keep log configuration.

In line 46 we’re calling run() function where actual business logic should be put. In this example, it just logs run().

I’ve skipped details description of init_app_conf.parse_config() function. I will start from init_app_conf.initConig() and I then I’ll describe init_app_conf.parse_config().

initConig() function can be optionally called prior any call to another function in this module. In the code snippet above I didn’t have such call. It is indented to be called in the MainThread. This function can be call with empty params. This function is idempotent. Parameters are :

  • default_parser_cls can be class or str. Optional. Default values is: AppConfParser This enables you to customize the parsing configuration logic for your needs.
  • default_parser_kwargs this params will be used as default values in default_parser_cls.__init__() function. Default values are , This means, by default converting config values is done using mask_value() function (see below).

parse_config() function. This is the main function of the module. I remind you, that ymlparsers and parser modules serves as Low-Level API for this module. So, if you want to change the default behavior, you should call ymlparsers.intConfig() first. See My ymlparsers module for more details.

What does this function do?

  • This function parses command line arguments first.
  • Than it parse yml files.
  • Command line arguments overrides yml files arguments.

Parameters of yml files we always try to convert on best-effort basses. Parameters of system args we try convert according to implicit_convert param.

In more detail, command line arguments of the form are parsed first. If exists it’s value is used to search for yml file. if is absent, ‘config.yml’ is used for yml file. If yml file is not found, only command line arguments are used. If yml file is found, both arguments are used, while command line arguments overrides yml arguments.

or appropriate key in default yml file is used to find ‘profiles’. Let suppose, that is resolved to config.yml. If ‘profiles’ is not empty, than it will be used to calculate filenames that will be used to override default yml file. Let suppose,’ resolved to [‘dev’, ‘local’] as in our code snippet. Than first ‘config.yml’ will be loaded, than it will be overridden with config-dev.yml, than it will be overridden with config-local.yml. At last, it will be overridden with system args. This entry can be always be overridden with system args.

general.whiteListSysOverride key in yml file is optional. If not provided, than any key that exists in the default yml file can be overridden with system args. If provided, than only key that start with one of the key provided here can be used to override entrys with system args. This entry can’t be overridden with system args.

or appropriate key in default yml file is used to instruct that listed key should be interpreted as comma-delimited list when is used to override entrys with system args. This entry can be always be overridden with system args.

general.config.file is key that is used in returned dict that points to default yml file.

If , than for system args we assume Python built-in types, including , and we're converting it to appropriate type.
Otherwise, will have the value that was set in initConig(). By default it is . See mask_value() function below.

Parameters:

  • . Can be . You can instantiate , for example, with default values. See here for example, or read documentation of here. Of course, you can just pass and make post-process for the populated dict.
  • : if not None, suppresses .
  • : if , than value that was passed to initConig() is used (default). if value attempt to convert value from system args to appropriate type will be done, ifvalue from system args will be used as is. See mask_value() function below.

mask_value() function implemented as a wrapper to method with support for variables. values are case-insensitive.

This function is used inside to get type for arguments that we get from system args.

This mechanism can be easily replaced with your own one, just provide , in initConig() function.

to_convex_map() utility function receives dictionary with flat keys, it has simple structure where value can’t be another dictionary. This method is not used in AlexBerUtils project. It will return dictionary of dictionaries with natural key mapping (see bellow), optionally, entries will be filtered out according to and, optionally, value will be implicitly converted to appropriate type.

In order to simulate dictionary of dictionaries flat keys compose key from outer dict with key from inner dict separated with dot. For example, general.profiles flat key corresponds to convex map with general key with dictionary as value that have one of the keys profiles with corresponding value.

If is , it will be used to filter out entries from the dict with flat keys.

If , than for system args we assume Python built-in types, including , and we're converting it to appropriate type.
Otherwise, will have the value that was set in initConig(). By default it is . See mask_value() function above.

Parameters:

  • dict with flat keys.
  • Optional. if present, only keys that start with one of the elements listed here will be considered.
  • : if , than value that was passed to initConig() is used (default). if value attempt to convert value from system args to appropriate type will be done, ifvalue from system args will be used as is. See mask_value() function above.

utility function merges value of 2 dicts. This value represents list of values. This function returns merged converted value, typically one from , if it is empty than from This function is used in module (see link below).

Parameters:

  • flat dictionary, usually one that was created from parsing system args.
  • dictionary of dictionaries, usually one that was created from parsing YAML file.
  • ] is absent or dict. See below.
  • is absent or list. See below.
  • : if , than value that was passed to initConig() is used (default). if value attempt to convert value from system args to appropriate type will be done, ifvalue from system args will be used as is. See mask_value() function above.

Value from is roughly obtained by .

Value from is roughly obtained by

If value (or intermediate value) is not found empty dict is used.

This method assumes that value contain that represent (comma-delimited).

This method assumes that contains . is applied only for .

If , than for system args we assume Python built-in types, including , and we're converting it to appropriate type.
Otherwise, will have the value that was set in initConig(). By default it is . See mask_value() function above.

This function returns merged converted value, typically one from , if it is empty than from

--

--

Senior Software Engineer at Pursway

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store