AcademySoftwareFoundation / AcademySoftwareFoundation/rez

rez-bind python erroneously set some PYTHONPATH for python/lib and python/extra

Open
#399 11 comments 0 reactions 0 assignees View on GitHub
bug rez-bind
Dominant language
Python
Stars
1.1k
Forks
369
Avg merge
12d 3h
Merged PRs (30d)
5

Description

Hi,

I discovered, already some days/week ago, that `rez-bind python` actually creates a package.py which contains instructions to add to PYTHONPATH the paths to 2 of the "main" libs directories of the python which was bound to. (it's the directories where the os and setuptools modules are available under the python which is bound to).
It (rez-bind) also has copied everything under these 2 paths into the package installation directory of that python.

We so end with :

`../packages/python/x.y.z/platform-linux/arch-x86_64/bin/python` -> link to the real python binary which was bound to.

But:

```
../packages/python/x.y.z/platform-linux/arch-x86_64/python/lib
../packages/python/x.y.z/platform-linux/arch-x86_64/python/extra
```

are now also in PYTHONPATH.

BUT, now anything which use a python under the hood/internally will have to use that PYTHONPATH containing directories to the selected (by rez-env) python own system libs/modules.

We can see that effect like this :

```
$ python -m site
sys.path = [
'/home/gregory',
'/home/gregory/rez/master-2.9.1/packages/python/2.6.6/platform-linux/arch-x86_64/os-CentOS-6.7/python/lib',
'/home/gregory/rez/master-2.9.1/packages/python/2.6.6/platform-linux/arch-x86_64/os-CentOS-6.7/python/extra',
'/usr/lib64/python26.zip',
'/usr/lib64/python2.6',
'/usr/lib64/python2.6/plat-linux2',
'/usr/lib64/python2.6/lib-tk',
'/usr/lib64/python2.6/lib-old',
'/usr/lib64/python2.6/lib-dynload',
'/home/gregory/.local/lib/python2.6/site-packages',
'/usr/lib64/python2.6/site-packages',
'/usr/lib64/python2.6/site-packages/PIL',
'/usr/lib64/python2.6/site-packages/gst-0.10',
'/usr/lib64/python2.6/site-packages/gtk-2.0',
'/usr/lib64/python2.6/site-packages/webkit-1.0',
'/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode',
'/usr/lib/python2.6/site-packages',
]
USER_BASE: '/home/gregory/.local' (exists)
USER_SITE: '/home/gregory/.local/lib/python2.6/site-packages' (exists)
ENABLE_USER_SITE: True
```

As you can see, given they are in PYTHONPATH, they take precedence over my .local (USER_SITE) directory.. which is a first not good.

But, if I would have rez-env with something using a python under hood, like nuke for instance, then Houston we have a (big) problem :

because nuke uses its own python, not necessarily compatible with the one/thoses I've bound to.

That is : nuke starts, it starts its own embedded python, which early import many important modules, like openssl, but then boum because the openssl for one python version isn't compatible with another. and given PYTHONPATH the python under nuke happens so to import the important modules from the python that was resolved in my rez-env call, not from his own python.

It seems to me you're trying, somehow, to do a _kind of_ virtualenv manually (but only a kind of) but the way it is actually triggers the problem(s) I mention / try to explain here. But PYTHONPATH is not done for that. well I mean putting a python system library/directory in PYTHONPATH is calling for problems.. as soon as you have to play with different python versions.

Actually, only doing the link to the python which is bound-to is enough to make it all available.

waiting answer / feedback or question ..

I can propose the PR (which is simply to not do the copy of these dirs and not add them to PYTHONPATH in the package.py which is generated) if you agree with this ..

Contributor guide

Open the contributing guide

Research direction

Start with the `rez-bind python` entry point and inspect how its generated `package.py` copies the Python `lib` and `extra` directories and adds them to `PYTHONPATH`. Reproduce the issue with `python -m site`; done means the bound executable remains available without injecting those system-library paths or copying them into the package installation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.