rust-lang / rust-lang/rust

no-op remap_path_prefix change invalidates incremental cache

Open
#132,132 7 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-debuginfo A-incr-comp A-path-remapping C-feature-request T-compiler WG-incr-comp
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I'm building rust code in a CI environment, setting -Cincremental to a shared location. CI allocates a new working directory for each build. To prevent leaking that info to the debug info, we set --remap_path_prefix=${PWD}= to eliminate the unwanted prefix from the debug info.

rustc src/lib.rs \
  --crate-name=xxx \
  --crate-type=rlib \
  --error-format=human \
  --out-dir=${OUTDIR} \
  --remap-path-prefix=${PWD}= \
  --emit=dep-info,link \
  -Ccodegen-units=256 \
  -Zincremental_info \
  -Cincremental=${CACHEDIR}

I expected this will reuse the incremental build cache, but it's not:

[incremental] session directory: 259 files hard-linked
[incremental] session directory: 0 files copied
[incremental] completely ignoring cache because of differing commandline arguments

Further inspection of how the hash being generated, I found:

https://github.com/rust-lang/rust/blob/017ae1b21f7be6dcdcfc95631e54bde806653a8a/compiler/rustc_session/src/options.rs#L195-L222

for our case, working_dir always uses remapped_path "", which is stable over build. but the remap_path_prefix is different everytime. and the logic for populating working_dir is at:

https://github.com/rust-lang/rust/blob/017ae1b21f7be6dcdcfc95631e54bde806653a8a/compiler/rustc_session/src/config.rs#L2690-L2695

So to get correct remapped working_dir, I must set --remap-path-prefix, but setting it will change remap_path_prefix and causing hash mismatch:

https://github.com/rust-lang/rust/blob/017ae1b21f7be6dcdcfc95631e54bde806653a8a/compiler/rustc_incremental/src/persist/load.rs#L159-L164

Meta

rustc --version --verbose:

rustc 1.82.0-nightly (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: aarch64-apple-darwin
release: 1.82.0-nightly
LLVM version: 19.1.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the option hashing in compiler/rustc_session/src/options.rs, the remapped working-directory setup in compiler/rustc_session/src/config.rs, and cache validation in compiler/rustc_incremental/src/persist/load.rs. Reproduce the shared incremental-cache build using changing --remap-path-prefix values, then trace which command-line inputs affect the hash. Done means a no-op remap change no longer prevents valid cache reuse, with regression coverage for the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.