PyO3 / PyO3/pyo3

Prevent pyo3 rebuilds with `cargo`, `maturin`, `rust-analyzer`, ... through generated build config file

Open
#5,439 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

When, in one crate, Python extension modules are build (with maturin), and Rust executables (with cargo), whenever launching the other build, pyo3 is rebuilt too.

This is triggered for me because maturin sets:

PYO3_ENVIRONMENT_SIGNATURE=cpython-3.13-64bit
PYO3_BUILD_EXTENSION_MODULE=1

But cargo build has both env vars unset. Same goes for IDE-run rust-analyzer, ... so it rebuilds 😢.

So this is a different take on #1708.

❓ Workaround: use something like direnv and put the env vars above in .envrc so they are set for cargo and rust-analyzer and won't change between runs (determine them with CARGO_LOG=cargo::core::compiler::fingerprint=info cargo build after running maturin).

⭐ Real solution proposal: Move away from the env-vars approach.

  • I think the current approach needs to be re-thought:
    • Pyo3 reconfiguration is only triggered by environment changes (rerun-if-env-changed), but by default the env is (and should be) empty => Using the environment as signaling method will never get rid of unintended rebuilds if not all tools set the exact same envirnment as signals as well.
  • How about using rerun-if-changed with files instead:
    • instead of syncing an environment variable name, use a common generated configuration file (e.g. $OUT_DIR/pyo3_buildconfig.toml?)
    • each tool aware of it, either adjusts it or leaves it unchanged if it discovered/decided on the same variable value
    • any tool invocation like rust-analyzer doesn't touch it anyway, so we don't have a rebuild

🟢 So in other words: write config file(s) instead of env vars, because they remain the same/unchanged if tools are not aware of them. Tools that aware of them are expected to adjust them, and then trigger rebuilds.

What do you think?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining how maturin, cargo, and rust-analyzer use PYO3_ENVIRONMENT_SIGNATURE and PYO3_BUILD_EXTENSION_MODULE during builds. Compare their rebuild behavior, then work out the generated configuration-file approach and its change conditions; done means unchanged invocations no longer rebuild pyo3 while intended configuration changes still do.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.