pypa / pypa/setuptools

pip 21.3+ in-tree-builds with setuptools, do not work with read-only source trees

Open
#3,237 21 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

Description

We are using pip as part of our build pipeline within a project containing a mix of languages. In order to keep everything clean and reproducible, we mount the source code read-only in a docker container and run the build steps there.

Even it's a mix of languages and tools, this approach used to work fine until now, when pip as the only of all the tools involved suddenly tried to touch the source tree during build.

I have read though the discussion in https://github.com/pypa/pip/issues/7555 that led to this change, but I feel like the point that there might be good reasons not to touch the source directory didn't get enough attention:

  • You want to keep source code and build artifacts separate
  • You don't want to clutter your scm
  • Files created within the docker container are usually owned by root. So in our setup, our source directory would be cluttered with files owned by root after the container exits
  • You want to make sure that no build script accidentally modifies the source code at build time. This is a very important point especially in large projects where it's hard to find side-effects in build scripts
Expected behavior

All other major packaging tools I know offer an option to chose a build directory that is separate from the source dir. I would expect that pip works in a similar way.

pip version

21.3

Python version

3.10

OS

ubuntu

How to Reproduce

Create a python package, mount read-only into a docker container and try to let pip install it, e.g:

mkdir readonly
echo "from setuptools import setup\nsetup(name='hello', version='1.0.0')" > readonly/setup.py
docker run -ti --rm -v `pwd`/readonly:/src:ro python:3.10 /bin/bash -c "pip install --upgrade pip; pip install /src"
rm -rf readonly
Output
Building wheels for collected packages: hello
  Building wheel for hello (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/src/setup.py'"'"'; __file__='"'"'/src/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gx0vxrjn
       cwd: /src/
  Complete output (8 lines):
  running bdist_wheel
  running build
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  creating hello.egg-info
  error: could not create 'hello.egg-info': Read-only file system
  ----------------------------------------
  ERROR: Failed building wheel for hello
  Running setup.py clean for hello
Failed to build hello
Code of Conduct

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the provided Docker command, using the read-only /src mount and setup.py example. Trace the setuptools egg_info step that attempts to create hello.egg-info; done means pip can install the package while keeping the source tree read-only and build artifacts separate.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.