indygreg / indygreg/PyOxidizer

No way to build packages with setup.py requirements

Open
#202 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

This is admittedly a non-standard use case, but it breaks PyOxidizer on our internal libraries. For integration purposes we have something like the following at the top of our internal library `setup.py` files:

```
from distgradle import GradleDistribution

setup(
distclass=GradleDistribution,
...
)
```

This works for us because our build system guarantees that `distgradle` will be installed in the venv before this `setup.py` is executed. However, this breaks PyOxidizer because even though I can add a `PipInstallSimple` rule to install `distgradle` before the `PipRequirementsFile` rule that pulls in all of our internal dependencies, IIUC PyOxidizer first builds dependencies in a temp venv because it might have to manipulate the built packages for static linking and such.

Thus `PipInstallSimple(package='/path/to/distgradle.whl')` will put distgradle in the target venv but it won't be available in the temp venv for our internal library build. For the same reason, adding `distgradle` to my `requirements.txt` also doesn't work.

While more rare, this can be a problem with PyPI packages, if they also have these kinds of implicit pre-install dependencies. I remember seeing such packages, but don't have an example handy. Another possible problem would be packages which build differently depending on what other packages are already installed (e.g. conditional dependencies, etc).

I was thinking about maybe adding a feature for "pre-installs" which would land in the temp venv before the rule would run. While ugly, it probably wouldn't be too terrible.

Another option would be to clone the target venv, or merge the target venv with the temp venv before you try to build the individual dependency. Merging and cloning venvs though is fraught with peril.

I'm open to other suggestions, and I'm motivated to help implement a solution (if it's not to complex).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the PipInstallSimple and PipRequirementsFile rules and how they create and populate the temporary build virtual environment. Compare the proposed pre-install, cloning, and merging approaches, then define a focused design and tests showing that a setup.py dependency such as distgradle is available while the internal package is built.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.