indygreg / indygreg/PyOxidizer

Consider rebuilding distributions when `pip` v21.3 comes out; "in-tree-build" notably faster

Open
#416 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

TL;DR - Passing `--use-feature=in-tree-build` to `pip` (via `export PIP_USE_FEATURE=in-tree-build` before `pyoxidizer build`) speeds up builds (especially if directory containing `setup.py` has many small files) because `pip` doesn't have to copy your local package to a temporary directory. This behavior will become the default in `pip` 21.3, so it would be nice to have that version bundled with the official `pyoxidizer` Python distributions.

Iterating on a `pyoxidizer.bzl` config, building frequently, each build took ~2m20s (using system Rust and already-cached PyPI packages) which felt like a pretty big speedbump. I noticed that it would often pause for a considerable amount of time on "Processing {path-to-local-package}". I already had my dependencies specified in a `setup.py` before adding pyoxidizer, so `exe.add_python_resources(exe.pip_install([CWD]))` felt most natural. I tried explicitly adding the dependencies explicitly in the `.bzl` instead, followed by an `exe.read_package_root` for the local package; maybe I should switch to a `requirements.txt` or something if there's significant savings? That dropped the build time to ~1m30s.

I noticed, though, that there was a deprecation notice right before the big "Processing" pause. It said that a future version of `pip` will build local packages in place without copying to a temporary directory, and that you can preview that behavior with `--use-feature=in-tree-build`. I tried that, with the original `exe.pip_install([CWD])` form, and it dropped the build time further to ~1m15s. (The local package is very simple, so it's possible that `setuptools` can make assumptions that the general-purpose `read_package_root` can't?) I suppose that the standard pip behavior wanted to copy the entire repo to a temporary folder, since `setup.py` is at the root, and that includes all my Rust, all my test data and other artifacts, _and_ the entire virtualenv I use for non-oxidized development and testing! That's silly, especially because there are a lot of small files, so I'm glad there's already a way to work around this.

I don't know how characteristic my setup is of `pyoxidizer` users, but I can imagine that it's one of the more ergonomic ways to oxidize an existing Python project that you control the source of.

I also don't know what the release cadence of the `pyoxidizer` Python distributions is, but I figured I should point this out on the off-chance you feel like updating them. Regardless, I hope this info can help some other people out!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the pyoxidizer.bzl usage of exe.pip_install([CWD]) and the process that builds the official PyOxidizer Python distributions. Check how the bundled pip version is selected, then determine whether pip 21.3 can be included and verify that in-tree builds work for this use case without regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, release
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.