PyO3 / PyO3/pyo3

Make pyo3-build-config more caching friendly

Open
#6,113 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

Problem

When a project is built with a PEP 517 frontend with build isolation (pip, uv, build), the frontend creates a randomly-named temporary environment and maturin points PyO3 at that environment's interpreter:

maturin pep517 build-wheel -i /home/runner/.cache/uv/builds-v0/.tmpyHvXPy/bin/python

pyo3-build-config's build script registers cargo:rerun-if-env-changed=PYO3_PYTHON. That path is inside the temporary virtualenv, so its different on every build, and cargo recompiles pyo3-build-config, pyo3-ffi, pyo3, pyo3-macros-backend, and pyo3-macros on every pip/uv install ., even when the interpreter, version, and ABI are identical and the cargo target dir is fully warm.

For any project that caches the target dir in CI (e.g. Swatinem/rust-cache), this defeats the cache. Measured on pyca/cryptography's CI: every run recompiled the five pyo3 crates on every platform; on Windows it added ~45s per job.

Proposed Solution

My proposal is two parts:

  • Change maturin to export a PYO3_BASE_PYTHON env var (in addition to PYO3_PYTHON) which points to the stable python binary path (i.e. the non-venv path, basically sys._base_executable)
  • Change pyo3-build-config to prefer reading PYO3_BASE_PYTHON if it's available

This should give you: pyo3-build-config has a stable path for metadata purposes, and PYO3_PYTHON is still available if you need it.

What do folks think? Should be easy to do PRs for this, and it should rollout fine:

maturin pyo3 behavior
old old status quo
new old extra env var ignored; status quo
old new variable absent; status quo
new new pyo3 chain stays cached across PEP 517 builds

cc @messense

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the pyo3-build-config build script's handling of PYO3_PYTHON and the maturin PEP 517 build path described in the issue. Check how the stable base interpreter path can be passed alongside the existing variable, then verify the old/new maturin and pyo3 combinations in the behavior matrix and confirm that repeated builds no longer invalidate the pyo3 chain unnecessarily.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.