Make pyo3-build-config more caching friendly
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.2k
- Forks
- 1k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 66
Description
Problem
When a project is built with a PEP 517 frontend with build isolation (pip, uv, build), the frontend creates a randomly-named temporary environment and maturin points PyO3 at that environment's interpreter:
maturin pep517 build-wheel -i /home/runner/.cache/uv/builds-v0/.tmpyHvXPy/bin/python
pyo3-build-config's build script registers cargo:rerun-if-env-changed=PYO3_PYTHON. That path is inside the temporary virtualenv, so its different on every build, and cargo recompiles pyo3-build-config, pyo3-ffi, pyo3, pyo3-macros-backend, and pyo3-macros on every pip/uv install ., even when the interpreter, version, and ABI are identical and the cargo target dir is fully warm.
For any project that caches the target dir in CI (e.g. Swatinem/rust-cache), this defeats the cache. Measured on pyca/cryptography's CI: every run recompiled the five pyo3 crates on every platform; on Windows it added ~45s per job.
Proposed Solution
My proposal is two parts:
- Change
maturinto export aPYO3_BASE_PYTHONenv var (in addition toPYO3_PYTHON) which points to the stable python binary path (i.e. the non-venv path, basicallysys._base_executable) - Change
pyo3-build-configto prefer readingPYO3_BASE_PYTHONif it's available
This should give you: pyo3-build-config has a stable path for metadata purposes, and PYO3_PYTHON is still available if you need it.
What do folks think? Should be easy to do PRs for this, and it should rollout fine:
| maturin | pyo3 | behavior |
|---|---|---|
| old | old | status quo |
| new | old | extra env var ignored; status quo |
| old | new | variable absent; status quo |
| new | new | pyo3 chain stays cached across PEP 517 builds |
cc @messense
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the pyo3-build-config build script's handling of PYO3_PYTHON and the maturin PEP 517 build path described in the issue. Check how the stable base interpreter path can be passed alongside the existing variable, then verify the old/new maturin and pyo3 combinations in the behavior matrix and confirm that repeated builds no longer invalidate the pyo3 chain unnecessarily.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100