AcademySoftwareFoundation / AcademySoftwareFoundation/rez

rez-pip release flag does not make use of release hooks

Open
#874 5 comments 1 reaction 0 assignees View on GitHub
bug rez-pip
Dominant language
Python
Stars
1.1k
Forks
374
Avg merge
9d 12h
Merged PRs (30d)
5

Description

After some investigation I was able to verify that the rez-pip command unfortunately does not call release hooks in the same fashion the rez-release command does.

Trying to release a normal package via rez-release calls configured release hooks as expected, this is done via the build process.

Please see the relevant excerpts below.

```py
# release.py
builder = create_build_process(opts.process,
working_dir,
package=package,
build_system=buildsys,
vcs=vcs,
ensure_latest=(not opts.no_latest),
skip_repo_errors=opts.skip_repo_errors,
ignore_existing_tag=opts.ignore_existing_tag,
verbose=True)
....

# perform the release
builder.release(release_message=release_msg or None,
variants=opts.variants){code}
```

```py
# build_process.py

def run_hooks(self, hook_event, **kwargs):
hook_names = self.package.config.release_hooks or []
hooks = create_release_hooks(hook_names, self.working_dir) for hook in hooks:
debug_print("Running %s hook '%s'...",
hook_event.label, hook.name())

....
```
```py
# local.py

def release(self, release_message=None, variants=None):
self._print_header("Releasing %s..." % self.package.qualified_name) # test that we're in a state to release
self.pre_release() release_path = self.package.config.release_packages_path
release_data = self.get_release_data()
changelog = release_data.get("changelog")
previous_version = release_data.get("previous_version")
previous_revision = release_data.get("previous_revision")

# run pre/post-release hooks
...
```

>❯ rez release
11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs...
11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.git...
11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.hg...
11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.stub...
11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.svn...
11:11:34 DEBUG Traceback (most recent call last):
File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/plugin_managers.py", line 140, in load_plugins
module = loader.find_module(modname).load_module(modname)
File "", line 462, in _check_name_wrapper
File "", line 962, in load_module
File "", line 787, in load_module
File "", line 265, in _load_module_shim
File "", line 702, in _load
File "", line 671, in _load_unlocked
File "", line 783, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs/svn.py", line 9, in
import pysvn
ModuleNotFoundError: No module named 'pysvn'

11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process...
11:11:34 DEBUG loading build_process plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process: rezplugins.build_process.local...
11:11:34 DEBUG loading build_process plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process: rezplugins.build_process.remote...
11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository...
11:11:34 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.filesystem...
11:11:34 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.memory...
11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system...
11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.bez...
11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.cmake...
11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.custom...
11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.make...
11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.rezconfig...
11:11:34 WARNING no 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.rezconfig
11:11:34 DEBUG Running command: /usr/bin/git rev-parse
**Releasing hello_world-1.0.0...**
Checking state of repository...
11:11:34 DEBUG Running command: /usr/bin/git rev-parse --is-bare-repository
11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@
Unknown macro: \{u}'
11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD
11:11:34 DEBUG Running command: /usr/bin/git diff-index --quiet HEAD
11:11:34 DEBUG command stdout:
11:11:34 DEBUG
11:11:34 DEBUG command stderr:
11:11:34 DEBUG
11:11:34 DEBUG Running command: /usr/bin/git diff-index --stat HEAD
11:11:34 DEBUG Running command: /usr/bin/git rev-parse HEAD
11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD
11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@'
11:11:34 DEBUG Running command: /usr/bin/git remote -v
11:11:34 DEBUG Running command: /usr/bin/git remote -v
11:11:34 DEBUG Running command: /usr/bin/git log
11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook...
11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.amqp...
11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.command...
11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.emailer...
*******************************Hook OK...************************
**11:11:34 DEBUG Running pre-release hook 'emailer'...**
Resolving build environment: python
resolved by lambdaclan@apollo, on Wed Apr 08 11:11:34 2020, using Rez v2.56.1

requested packages:
python
~platform==linux (implicit)
~arch==x86_64 (implicit)
~os==Arch-rolling (implicit)

resolved packages:
arch-x86_64 /home/lambdaclan/int/arch/x86_64
os-Arch-rolling /home/lambdaclan/int/os/Arch-rolling
platform-linux /home/lambdaclan/int/platform/linux
python-3.8.2 /home/lambdaclan/int/python/3.8.2/platform-linux/arch-x86_64/os-Arch-rolling

Invoking cmake build system...
11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.bash...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd...
11:11:34 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd did not return a class.
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.csh...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell...
11:11:34 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell did not return a class.
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.pwsh...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.sh...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.tcsh...
11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.zsh...
Executing: /bin/cmake -d /home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world -Wno-dev -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE -D_ECLIPSE_VERSION=4.3 --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX=/home/lambdaclan/releases/hello_world/1.0.0 -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -DCMAKE_BUILD_TYPE=Release -DREZ_BUILD_TYPE=central -DREZ_BUILD_INSTALL=1 -G Unix Makefiles -DCENTRAL=1
Not searching for unused variables given on the command line.
– The C compiler identification is GNU 9.3.0
– The CXX compiler identification is GNU 9.3.0
– Check for working C compiler: /bin/cc
– Check for working C compiler: /bin/cc - works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /bin/c++
– Check for working CXX compiler: /bin/c++ - works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found PkgConfig: /bin/pkg-config (found version "1.6.3")
– Configuring done
– Generating done
– Build files have been written to: /home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build

Executing: make -j4
make[1]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
Scanning dependencies of target py
make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
[100%] Generating py/python/hello_world.pyc
make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
[100%] Built target py
make[1]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'

Executing: make -j4 install
make[1]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
[100%] Built target py
make[1]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'
Install the project...
– Install configuration: "Release"
– Installing: /home/lambdaclan/releases/hello_world/1.0.0/./python/hello_world.py
– Installing: /home/lambdaclan/releases/hello_world/1.0.0/./python/hello_world.pyc
– Installing: /home/lambdaclan/releases/hello_world/1.0.0/./bin/hello
11:11:38 DEBUG Running command: /usr/bin/git rev-parse HEAD
11:11:38 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD
11:11:38 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@
Unknown macro: \{u}
'
11:11:38 DEBUG Running command: /usr/bin/git remote -v
11:11:38 DEBUG Running command: /usr/bin/git remote -v
11:11:38 DEBUG Running command: /usr/bin/git log
Hook OK...
**********************11:11:38 DEBUG Running post-release hook 'emailer'...***********************
Sending release email to:
- lambdaclan@myemail.com
11:11:38 ERROR release email delivery failed: [Errno -2] Name or service not known
11:11:38 DEBUG Running command: /usr/bin/git tag
**Release Summary**

1 of 1 releases were successful

Whereas rez-pip does not use the same build process as such no release hook seems to be ever called

> ❯ rez pip -i -r yapf
11:15:05 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository...
11:15:05 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.filesystem...
11:15:05 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.memory...
11:15:05 INFO Trying to use pip from python package
11:15:05 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.bash...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd...
11:15:05 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd did not return a class.
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.csh...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell...
11:15:05 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell did not return a class.
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.pwsh...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.sh...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.tcsh...
11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.zsh...
11:15:06 INFO Found pip-20.0.2 inside /home/lambdaclan/packages/python/3.8.2/package.py. Will use it with /bin/python3
11:15:06 INFO Installing 'yapf' with pip taken from '/bin/python3'
11:15:06 DEBUG

package download environment:
resolved by lambdaclan@apollo, on Wed Apr 08 11:15:05 2020, using Rez v2.56.1

requested packages:
python-3.8
~platform==linux (implicit)
~arch==x86_64 (implicit)
~os==Arch-rolling (implicit)

resolved packages:
arch-x86_64 /home/lambdaclan/packages/arch/x86_64 (local)
os-Arch-rolling /home/lambdaclan/packages/os/Arch-rolling (local)
platform-linux /home/lambdaclan/packages/platform/linux (local)
python-3.8.2 /home/lambdaclan/packages/python/3.8.2/platform-linux/arch-x86_64/os-Arch-rolling (local)

11:15:06 DEBUG running: /bin/python3 -m pip install --use-pep517 --target=/tmp/pip-hpxiaohz-rez yapf
╭─ ~/Workspace/GitHub/rez/example_packages/hello_world
╰─❯ 11:15:08 DEBUG Found /tmp/pip-hpxiaohz-rez/yapf-0.29.0.dist-info
11:15:08 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']}
11:15:08 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']}
11:15:08 DEBUG Pip to rez requirements translation information for yapf (0.29.0):
{'pip': {'run_requires': },
'rez': {'metadata': {'is_pure_python': True},
'requires': [],
'variant_requires': ['python-3.8']}}
11:15:08 INFO Installed [yapf-0.29.0] /home/lambdaclan/releases/yapf/0.29.0/package.py (11141ce2a13f3ba8344491dbd6c536267b83e386)
11:15:08 INFO 1 packages were installed.

The only thing that seems to change when passing the release flag (-r) to rez pip is the location at which the package is being stored:

```py
# src/rez/pip.py
if prefix is not None:
packages_path = prefix
else:
packages_path = (config.release_packages_path if release
else config.local_packages_path)
```

rez pip does not make use of the local build process which in turn calls the release hooks. It uses make package

```py
# src/rez/pip.py
# create the rez package
name = pip_to_rez_package_name(distribution.name)
version = pip_to_rez_version(distribution.version)
requires = rez_requires["requires"]
variant_requires = rez_requires["variant_requires"]
metadata = rez_requires["metadata"]

with make_package(name, packages_path, make_root=make_root) as pkg:
# basics (version etc)
pkg.version = version
.....
```

Since release hooks can be very useful for all kinds of automation tasks such as CI we will need to add proper release hook support to rez pip command.

The two options proposed were the following:

a. Make rez-pip command use a custom build system instead of custom package building

b. Just load and fire the hooks directly using the existing functions

After a discussion with @nerdvegas I had on Slack, it seems like the second option is preferred at least for now.

> a build system for pip would be something quite different to what rez-pip is currently doing (not incorrect, just different). Imo we should look at what it would take to call the hooks directly.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the release flow shown in release.py and build_process.py with the release implementation in local.py. Trace how rez-pip handles its release flag and where release hooks are invoked, then verify that its behavior matches rez-release without breaking the existing pre/post-release hook flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.