pypa / pypa/setuptools

[BUG] easy_install doesn't handle sysconfig vars containing multiple space separated paths

Open
#4,059 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Needs Repro Needs Triage
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

setuptools version

setuptools==68.2.2

Python version

Python3.11 (via pyenv)

OS

macos

Additional environment information
$ python3 --version
Python 3.11.2
$ python3 -m pip --version
pip 23.2.1 from /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/site-packages/pip (python 3.11)
Description

I'm using python3.11 via pyenv (this is the only version of python I have apart from system python3.9).
While I've never encountered this issue before, and it involves pyenv, I debugged a bit and I believe it's a setuptools rather than pyenv issue.

When trying to install any editable package, e.g. python3 -m pip install --user -e /path/to/pkg, somehow it creates an extra directory ending with a space in ~/.pyenv/versions:

$ ls -1 ~/.pyenv/versions/
3.11.2
3.11.2

(hard to see obviously because it ends with space 🙃 )

I think this causes some issues further down the line for pyenv, but it's kinda irrelevant for this bug.
I tried making ~/.pyenv/versions directory read only and running setuptools with debug information:

$ DISTUTILS_DEBUG=true python3 -m pip install --user -e /path/to/pkg
...
        Distribution.get_command_obj(): creating 'egg_info' command object
        os.makedirs('/Users/karlicos/.pyenv/versions/3.11.2 /Users/karlicos/.pyenv/versions/3.11.2/lib /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/lib-dynload', 0o700)
        error: [Errno 13] Permission denied: '/Users/karlicos/.pyenv/versions/3.11.2 '
...

It seems that this happens here https://github.com/pypa/setuptools/blob/82f5c60762eb16d9da9b19dbfc38b497a2939716/setuptools/command/easy_install.py#L1366-L1374

The offending variable turned out to be DESTDIRS, which contained a bunch of paths separated by a space. There is also INCLDIRSTROMAKE in my sysconfig that contains multiple paths

python3 -c 'import sysconfig; print({k: v for k, v in sysconfig.get_config_vars().items() if isinstance(v, str) and v.startswith("/Users/karlicos/") and " " in v})'
{'DESTDIRS': '/Users/karlicos/.pyenv/versions/3.11.2 /Users/karlicos/.pyenv/versions/3.11.2/lib /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/lib-dynload', 'INCLDIRSTOMAKE': '/Users/karlicos/.pyenv/versions/3.11.2/include /Users/karlicos/.pyenv/versions/3.11.2/include /Users/karlicos/.pyenv/versions/3.11.2/include/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/include/python3.11'}

I'm not sure what exactly the aim of this function, but perhaps it needs to do some extra checks before trying to create dirs. I suppose it's possible that it's a misconfiguration on my side somehow, but I think the variable names suggest they may contain multiple paths.

possibly relevant issue: https://github.com/pypa/setuptools/issues/3063

Expected behavior

Package is installed as editable, no extra dirs are created

How to Reproduce

Not sure how to provide consistent steps to reproduce since it requires pyenv set up in the first place, and there is no Docker on macos to try it

Output

provided above

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at setuptools/command/easy_install.py around lines 1366-1374 and inspect how sysconfig variables such as DESTDIRS and INCLDIRSTOMAKE are handled. Reproduce with the reported editable-install command and DISTUTILS_DEBUG=true where possible; done means an editable package install does not create directories from space-separated path values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.