AcademySoftwareFoundation / AcademySoftwareFoundation/rez
Allow rez.cli to be used as a high level python API
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 369
- Avg merge
- 12d 3h
- Merged PRs (30d)
- 5
Description
This might not make the cut for 2.0, but I wanted to make a ticket for it while I remember.
It would be nice to have a high-level python API which matches exactly the rez command-line interface (i.e. is auto-generated from it, or vice versa). This would make it exceedingly easy for a developer to access high-level functionality without needing to learn rez's internal API. We could present the CLI API as the stable, future-proof API, thus allowing the lower-level API to make breaking changes with major releases.
Some notes on the current rez command-line interface:
- it is implemented using `argparse`
- each sub-command is stored in its own submodule ( `rez env` in `rez.cli.env`, `rez release` in `rez.cli.release`, etc)
- a sub-command module must contain a `__doc__` attribute, a `setup_parser` function, and a `command` function.
- command sub-modules are imported lazily. e.g. `rez.cli.env` is only imported if a user invokes `rez env` or `rez -h` (which imports all sub-modules).
Here are two takes on how to achieve the goal of this ticket. the first creates the API from the existing CLI, and the second creates the CLI from an API. Ultimately, the latter is the better solution, but I will explain both for the sake of completeness.
**Option 1: API wraps CLI**
**overview:** create functions that wrap the existing cli `command` functions. e.g. `rez.cmds.release` would wrap `rez.cli.release.command`.
To implement, we would first create a new module to house the cli api called `rez.cmds` (or `rez.api`). on import of `rez.cmds` the module would import all of the `rez.cli` submodules, and create functions out of each of their `command` functions.
For example, consider `rez.cli.settings`:
``` python
def setup_parser(parser):
parser.add_argument("-p", "--param", type=str,
help="print only the value of a specific parameter")
parser.add_argument("--pp", "--packages-path", dest="pkgs_path", action="store_true",
help="print the package search path, including any "
"system paths")
def command(opts, parser):
...
```
on import, `rez.cmds` would import `rez.cli.settings` (and all other cli sub-mdoules), execute the `setup_parser` function, and use the information on the returned parser object to generate a wrapper function `rez.cmds.settings` which accepts 2 arguments: `param` and `packages_path`. at runtime, the wrapper would handle default values, convert the arguments to an `argparse.Namespace` instance and pass that to the `opts` argument of the `command` function.
we would have to do away with the `parser` argument to the `command` functions, but as far as I can tell this is used only by `parse_build_args`, and just to call `parser.error`, which I'm sure we can handle another way.
**Option 2: CLI wraps API**
**Overview:** use [click](http://click.pocoo.org/)'s decorator approach to create functions which can double as a CLI and an API.
[click](http://click.pocoo.org/) is a new module for creating command-line interfaces (from the author of flask). One of the 3 features touted in its docs is that it "supports lazy loading of subcommands at runtime". Lazy-loading of individual command sub-modules is one of the key features of rez's current cli design, however forcing `argparse` to load sub-commands lazily took quite a bit of work (which you'll find in `rez.cli._main` in the form of `LazySubParsersAction`, `LazyArgumentParser`, and `SetupRezSubParser`). I found out about click shortly after making these lazy-loading changes. I think the code would be much simpler using `click`.
A basic CLI built with `click` looks like this:
``` python
import click
@click.group()
def cli():
pass
@cli.command()
@click.option('--port', default=8000)
def runserver(port):
click.echo('Serving on http://127.0.0.1:%d/' % port)
if __name__ == '__main__':
cli()
```
assuming this was the contents of a file called`myserver`, you could then call `myservice runserver --port=5000`.
Since click commands are normal functions with named arguments, which are then decorated to expose in the CLI, it struck me that the original functions could double as a python API. I started playing around with this idea and had some initial success. I created an issue to the effect [here](https://github.com/mitsuhiko/click/issues/40#issuecomment-41628939). While I failed to convince the author to add native support for this concept to click, my experimentation showed that it is pretty easy to add ourselves (much easier than the hacking I had to do to get `argparse` to lazily load sub-commands`)
In order for this plan to work cleanly, I would also like to look into whether `click.option` can automatically grab defaults from the function itself. e.g.:
``` python
@cli.command()
@click.option('--port') # implicit `default=8000`
def runserver(port=8000):
click.echo('Serving on http://127.0.0.1:%d/' % port)
```
We might need to edit click source code and submit a pull request to implement this.
As to how to organize this code to keep command-line performance snappy, I'm not completely decided. we could either:
1. place undecorated command functions in `rez.cmds`, and import this module into each sub-module to decorate its respective function. this would ensure that decoration code only runs if the sub-command is invoked, but we'd have to be careful that functions in `rez.cmds` continue to keep their imports local since this same module would be imported by all sub-commands. this has the advantage of presenting a single module containing the entire high-level python API which developers can easily peruse. the disadvantage is that the highly interrelated actions of declaring a function and decorating it is split between two files
2. place decorated command functions in their respective sub-modules. `rez.cmds` would import those sub-modules to get access to the original undecorated functions. the primary disadvantage here is that the python API in `rez.cmds` is dynamically generated from `rez.cli` and thus would be slightly more confusing for developers looking through the code to comprehend. could be solved with a good set of sphinx docs.
3. don't bother with `rez.cmds`. instead put _decorated_ functions into `rez.cli`, have this double as the api, and test the performance impact of doing so. it could be offset by removing the sub-module import.
Contributor guide
Research direction
Start by reading the command structure in rez.cli and the lazy-loading implementation in rez.cli._main, including LazySubParsersAction, LazyArgumentParser, and SetupRezSubParser. Compare the proposed rez.cmds wrapper approach with the click-based alternative, then define the API and CLI behavior that must match before implementing it. Done means a documented high-level Python API with corresponding command coverage and tests for argument handling and lazy loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100