Config utilities for describing config files and CLI arguments #17

NikolausDemmel · 2014-07-29T20:37:50Z

new module config_utils for a general purpose tool to describe config files, parse
yaml config files and expose specific arguments to the command line; includes some
unit tests for the config item types
new module config describes xylem's global configuration; includes the logic to
compute sources and cache folders given command line arguments and config files
new module arguments for global command line arguments (moved from utils);
interacts with config to include and process global config arguments
new module yaml_utils contains customization to loading and dumping yaml (moved
(from utils)
--help argument now uses pydoc.pager to display the help in a less-like scrollable
manner on pty's
help output includes new section config arguments, describing the arguments
that correspond to a config file entry
commands have a new hook to add to the config description; some logic common
to all commands was moved to commands.main
context objects like InstallerContext and SourcesContext now take a config
dictionary as argument, which defaults to the global config singleton if not
passed explicitly
high-level API used by the commands (update, resolve, lookup) is also
parameterized by an optional config dict argument, which defaults to the
global config
Introduce XylemError as common Exception base class. Having a common
base class allows users to catch all xylem related exceptions in one handler.

Output of xylem resolve --help:

usage: xylem resolve [--show-trumped] [--all] [--show-priority]
                     [--show-default-installer] [--os name:version]
                     [--os-features "feature1,feature2,..."]
                     [--core-installers "inst1,inst2,..."]
                     [--use-additional-installers yes|no]
                     [--install-from "inst1:[key1,key2,...] ..."]
                     [--installer-options "inst1:{opt1:val1,...}, ..."]
                     [--user-sources yes|no] [--cache-dir PATH]
                     [--sources-dir PATH] [-h] [--version] [-v] [-d]
                     [--print-config] [--no-user-config]
                     [--prefix XYLEM_PREFIX] [--xylem-dir XYLEM_DIR]
                     [xylem_key [xylem_key ...]]

Lookup a xylem key and resolve to unique, parsed rule based on priorities.

positional arguments:
  xylem_key

optional arguments:
  --show-trumped        Show all possible resolutions for key, also for
                        trumped installers.
  --all                 Resolve all keys with resolution for this OS.
  --show-priority       Show priority of installer.
  --show-default-installer
                        Show installer even if it is the default installer.

config arguments:
  The following typed arguments correspond to entries in the config file.
  Command line argument values are interpreted as YAML and override the
  corresponding entries in user/system config files.

  There are some special cases and short hand notation for parsing config
  arguments:
  * List: the outer '[]' can be omitted; multiple occurrences of the same
  command line argument are concatenated
  * MergingDict: the outer '{}' can be omitted; multiple occurrences of the
  same command line argument are merged; value is merged with the config
  file entries
  * Path: string parsed as-is; path is expanded (e.g. `~`)
  * String: string parsed as-is

  --os name:version     override os detection; if no ':' is present, the
                        entire string is interpreted as the os name and the
                        version is detected
                        (config: `os_override`, type: String)
  --os-features "feature1,feature2,..."
                        OS features to be used. The list of possible values
                        and the default choice is defined by the os plugin for
                        the selected OS.
                        (config: `os_options/features`, type: List)
  --core-installers "inst1,inst2,..."
                        core installers to be used. The default list is
                        defined by the OS plugin for the selected OS.
                        Additional installers as defined by installer plugins
                        may be used on top of the core installers unless the
                        `use_additional_installers` option is set to False.
                        (config: `os_options/installers`, type: List)
  --use-additional-installers yes|no
                        if `True`, use additional installers as defined by
                        installer plugins on top of the core installers
                        (type: Boolean, default: True)
  --install-from "inst1:[key1,key2,...] ..."
                        mapping installer names to list of xylem keys;
                        overwrites the installer priority such that the given
                        keys are only ever installed with the specified
                        installer
                        (type: MergingDict)
  --installer-options "inst1:{opt1:val1,...}, ..."
                        options passed to installer plugins; valid options are
                        specific to each installer plugin
                        (type: MergingDict)
  --user-sources yes|no
                        if `True`, look for sources and cache in user
                        directory instead of system-wide locations
                        (type: Boolean, default: False)
  --cache-dir PATH      override the cache location
                        (type: Path)
  --sources-dir PATH    override the sources directory
                        (type: Path)

global arguments:
  By default xylem operates with system-wide configuration of sources and
  cache. The path under which those are found can be set with the
  XYLEM_PREFIX environment variable (or `--prefix` argument). To instead use
  sources and directories in the user's home directory, make use of the
  `--user- sources` argument. An alternative mode of configuration is by
  setting the XYLEM_DIR environment variable (or `--xylem-dir` argument).
  With that, only config files in that directory are used and sources/cache
  folder is local to the XYLEM_DIR. This can be used by third party tools to
  utilize xylem with config/sources/cache in a temporary directory.

  -h, --help            show this help message and exit
  --version             print xylem version and exit
  -v, --verbose         verbose console output
  -d, --debug           enable debug messages (overwrites the XYLEM_DEBUG
                        environment variable)
  --print-config        print the effective configuration (from command line
                        and config files) and exit
  --no-user-config      disable user config (system-wide config only)
  --prefix XYLEM_PREFIX
                        Set the FHS prefix for finding system-wide
                        configurations and caches. System-wide configuration
                        files will be looked for in 'XYLEM_PREFIX/etc/xylem'
                        and cache files will be placed in
                        'XYLEM_PREFIX/var/cache/xylem'. The default prefix is
                        '/'. The prefix can also be set by the XYLEM_PREFIX
                        environment variable, but the command line option
                        takes precedence.
  --xylem-dir XYLEM_DIR
                        Specify the path of directory to use for configuration
                        and caches. Configuration files will be looked for in
                        'XYLEM_DIR' and caches will be placed in
                        'XYLEM_DIR/cache'. This is intended to be used by
                        tools that want to create a local configuration and
                        cache independent from the system-wide setup. If set,
                        only the configuration files in XYLEM_DIR are used
                        instead of system and user configuration files, i.e.
                        XYLEM_PREFIX will be ignored. This can also be set by
                        the XYLEM_DIR environment variable, but the command
                        line option takes precedence.

Given the following config file in ~/.config/xylem/config.yaml,

install_from:
  homebrew: [foo, bar, baz]
installer_options:
  apt:
    repositories: [{sourceline: "foo bar"}]
os_options:
  features: []
os_override: osx
user_sources: yes

the output of two example commands to illustrate the config functionality follows:

$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --print-config
cache_dir: /Users/demmeln/.config/xylem/cache
install_from:
  apt: [foo, bar]
  homebrew: [foo, bar, baz]
  pip: [quux, quax]
installer_options:
  apt:
    repositories: [{sourceline: foo bar}]
os_options:
  features: [python3]
  installers:
os_override: [ubuntu, trusty]
sources_dir: /Users/demmeln/.config/xylem/sources.d
use_additional_installers: true
user_sources: true

$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --debug --verbose
overriding OS to [ubuntu:trusty]
initializing database with sources dir `/Users/demmeln/.config/xylem/sources.d` and cache dir `/Users/demmeln/.config/xylem/cache/sources`
No configs found in configured source dir '/Users/demmeln/.config/xylem/sources.d', using default sources.
Loading default source urls
boost --> libboost-all-dev

Having a common base class allows users to catch all xylem related exceptions in one handler.

- extensive tests for config types - tests for other parts of config utils are still TODO

- new module `config` describes the global configuration - `config` includes the logic to compute soruces and cache folders given command line arguments and options - `arguments` module features help action, which uses pydoc.pager to display the help in a `less`-like scrollable manner on pty's - help output includes new section `config arguments`, describing the arguments that correspond to a config file entry - commands have a new hook to add to the config description; some logic common to all commands was moved to `commands.main` - context objects like `InstallerContext` and `SourcesContext` now take a config dictionary as argument, which defaults to the global config sigleton if not passed explicitly - high-level API used by the commands (`update`, `resolve`, `lookup`) is also parameterized by an optional config dict argument, which defaults to the global config

NikolausDemmel · 2014-07-29T20:38:55Z

@wjwwood please review.

This has become a bit more extensive than anticipated. More stuff to follow soon.

wjwwood · 2014-07-29T22:49:20Z

@NikolausDemmel Extensive indeed, I'll do my best to review it tonight.

wjwwood · 2014-08-11T02:28:37Z

xylem/config_utils.py

+    # add argument groups and some global config-related arguments that
+    # are independent from the description
+    group = parser.add_argument_group('global arguments')
+    add_global_config_arguments(group, "footool")


Should this also use TOOL_NAME?

wjwwood · 2014-08-11T02:40:43Z

I'm not sure if this is desired, but I get this when using the --print-config global option:

% xylem --print-config
usage: xylem [--os name:version] [--os-features "feature1,feature2,..."]
             [--core-installers "inst1,inst2,..."]
             [--use-additional-installers yes|no]
             [--install-from "inst1:[key1,key2,...] ..."]
             [--installer-options "inst1:{opt1:val1,...}, ..."]
             [--user-sources yes|no] [--cache-dir PATH] [--sources-dir PATH]
             [-h] [--version] [-v] [-d] [--print-config] [--no-user-config]
             [--prefix XYLEM_PREFIX] [--xylem-dir XYLEM_DIR]
             [resolve | lookup | update] ...
xylem: error: too few arguments

wjwwood · 2014-08-11T02:46:22Z

When I run xylem resolve with no arguments, nothing happens, I would expect usage for resolve:

% xylem resolve

wjwwood · 2014-08-11T02:58:29Z

Ok, other than the few small comments this is a +1 from me. Please fixup those things I touched on, merge this, and then rebase the other pull requests.

NikolausDemmel · 2014-08-11T11:27:59Z

I'm not sure if this is desired, but I get this when using the --print-config global option:

It seems to be a limitation of argparse to not allow optional subparsers. The simplest fix might involve http://stackoverflow.com/a/22054398/1813258.

P.S.: I just stumbled upon http://docopt.org/ (the recorded talk is a nice watch). I seems like a really powerful and simple way to do CLI, but unfortunately it looks to be not as practical when options come from different places.

When I run xylem resolve with no arguments, nothing happens

Will fix.

Config utilities for describing config files and CLI arguments - new module `config_utils` for a general purpose tool to describe config files, parse yaml config files and expose specific arguments to the command line; includes some unit tests for the config item types - new module `config` describes xylem's global configuration; includes the logic to compute sources and cache folders given command line arguments and config files - new module `arguments` for global command line arguments (moved from `utils`); interacts with config to include and process global config arguments - new module `yaml_utils` contains customization to loading and dumping yaml (moved (from `utils`) - `--help` argument now uses pydoc.pager to display the help in a `less`-like scrollable manner on pty's - help output includes new section `config arguments`, describing the arguments that correspond to a config file entry - commands have a new hook to add to the config description; some logic common to all commands was moved to `commands.main` - context objects like `InstallerContext` and `SourcesContext` now take a config dictionary as argument, which defaults to the global config singleton if not passed explicitly - high-level API used by the commands (`update`, `resolve`, `lookup`) is also parameterized by an optional config dict argument, which defaults to the global config - Introduce XylemError as common Exception base class. Having a common base class allows users to catch all xylem related exceptions in one handler.

NikolausDemmel · 2014-08-11T12:02:34Z

Ok, other than the few small comments this is a +1 from me. Please fixup those things I touched on, merge this, and then rebase the other pull requests.

args/opts is pending a decision
for xylem --print-config I opened Fix xylem --print-config #22

I addressed the other comments as a new commit 720d3db ontop of the most recent stuff to avoid rebasing so many commits and possibly manually mergin a bunch of times.

Thanks for the comments.

NikolausDemmel added 13 commits July 24, 2014 02:24

Move global argument code to seperate file.

6f09186

Introduce XylemError as common Exception base class.

5c10889

Having a common base class allows users to catch all xylem related exceptions in one handler.

Somewhat general purpose config_util module.

770ec03

Unit tests; new test dependecy testfixtures

3979262

- extensive tests for config types - tests for other parts of config utils are still TODO

fix doc generation warning

fb3b4b0

fix docstring typos

b067cfa

update apidoc

9246fe6

Move yaml utilities to new yaml_utils; support dumping ConfigDict

ee91f5b

Fix outdated imports.

ccbe973

Fix --os-override parsing.

52433fb

help output adjustments

05483a2

fix pep8

94f3b68

NikolausDemmel mentioned this pull request Jul 30, 2014

Small util fixes and updated example sources #18

Closed

NikolausDemmel added the in progress label Aug 10, 2014

NikolausDemmel self-assigned this Aug 10, 2014

NikolausDemmel added this to the 0.1.0 Release milestone Aug 10, 2014

wjwwood reviewed Aug 11, 2014
View reviewed changes

NikolausDemmel added a commit that referenced this pull request Aug 11, 2014

small fixes from feedback in #17

720d3db

NikolausDemmel merged commit 7bc9791 into master Aug 11, 2014

NikolausDemmel mentioned this pull request Aug 11, 2014

Fix xylem --print-config #22

Open

NikolausDemmel added a commit that referenced this pull request Aug 11, 2014

small fixes from feedback in #17

6f096b8

NikolausDemmel added a commit that referenced this pull request Aug 11, 2014

small fixes from feedback in #17

6e56315

NikolausDemmel deleted the config branch August 11, 2014 12:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Config utilities for describing config files and CLI arguments #17

Config utilities for describing config files and CLI arguments #17

NikolausDemmel commented Jul 29, 2014

NikolausDemmel commented Jul 29, 2014

wjwwood commented Jul 29, 2014

wjwwood Aug 11, 2014

NikolausDemmel Aug 11, 2014

wjwwood commented Aug 11, 2014

wjwwood commented Aug 11, 2014

wjwwood commented Aug 11, 2014

NikolausDemmel commented Aug 11, 2014

NikolausDemmel commented Aug 11, 2014

Config utilities for describing config files and CLI arguments #17

Config utilities for describing config files and CLI arguments #17

Conversation

NikolausDemmel commented Jul 29, 2014

NikolausDemmel commented Jul 29, 2014

wjwwood commented Jul 29, 2014

wjwwood Aug 11, 2014

Choose a reason for hiding this comment

NikolausDemmel Aug 11, 2014

Choose a reason for hiding this comment

wjwwood commented Aug 11, 2014

wjwwood commented Aug 11, 2014

wjwwood commented Aug 11, 2014

NikolausDemmel commented Aug 11, 2014

NikolausDemmel commented Aug 11, 2014