`SCCACHE_BASEDIRS` + Preporcessor cache mode hands out another checkout's object file

Task: preprocessor cache mode hands out other checkouts' object files

Working file, do not commit it. Read AGENTS.md first; everything in it applies (tests are mandatory, PR descriptions stay short, comments explain why).

Summary

With SCCACHE_BASEDIRS configured and preprocessor cache mode enabled (the default for the disk cache), sccache silently returns an object file compiled from a different checkout of the tree. It is a wrong-code bug, not a performance one: the object links, and the program crashes later on a struct layout that no source file in the tree agrees with.

Reported against 0.18.0 on a repo with four git worktrees under /home/user/project/{wt1,wt2,wt3,wt4}, all listed as basedirs. Unit tests built in wt3 contained objects compiled in wt1 and wt2, which still used the previous layout of a class whose definition wt3 had changed, so the first virtual call through it jumped through a garbage pointer.

Mechanism

preprocessor_cache_entry_hash_key (src/compiler/preprocessor_cache.rs:376) hashes no file content at all - compiler digest, arguments, env, and the input file path. The manifest it keys stores every include as an absolute path (IncludeEntry::path), and PreprocessorCacheEntry::result_matches (src/compiler/preprocessor_cache.rs:194) stats and hashes exactly those paths. On a match, generate_hash_key returns the object cache key stored in the entry and never computes the content-based one (src/compiler/c.rs:508-521).

Basedirs exist to make two checkouts agree on a key. The moment they do, that validation is vacuous: wt3's compilation finds the manifest written by wt1, verifies wt1's headers against wt1's own files - which of course still match, they are what produced the cached object - and is handed wt1's object key. Nothing ever looks at wt3's headers.

So the entry is keyed by a path that basedirs made tree-independent, but its contents describe one specific tree. Those two cannot both be true.

Why 0.18.0 is where it started

The input path has been stripped since SCCACHE_BASEDIRS landed (#2521, 0.14), but the arguments were hashed verbatim - for arg in arguments at preprocessor_cache.rs:394 in v0.16.0 and v0.17.0. A CMake command line spells the checkout out several times:

/usr/bin/clang++ -isystem /home/user/project/wt3/generated/... \
    -c /home/user/project/wt3/generated/lib/widget/widget.cpp

so two worktrees could never share a preprocessor cache key, whatever the basedirs said. #2840 (0.18.0) strips basedirs from the arguments too, the last thing keeping the keys apart goes away, and they collapse. #2840 is the trigger, not the defect - the defect is that the manifest was never basedir-safe.

It reproduces on 0.17.0 and earlier for builds that pass relative arguments; see the third row of the table below. 0.18.0 extended it to every absolute-path build, which is to say every CMake project.

Unrelated to this: #2843 (assembler in the key) and #2824 (#line parsing).

Reproduction

Two trees differing only in a header, both listed as basedirs. offsetof(S, b) is 4 in treeA and 8 in treeB; treeB's object reports 4.

#!/usr/bin/env bash
set -e
R=$(mktemp -d); mkdir -p "$R/treeA/inc" "$R/treeB/inc" "$R/cache"
echo 'struct S { int a; int b; };'             > "$R/treeA/inc/cfg.h"
echo 'struct S { int a; int pad_x; int b; };'  > "$R/treeB/inc/cfg.h"
cat > "$R/treeA/main.cpp" <<'EOF'
#include "cfg.h"
int offset_of_b() { return (int)__builtin_offsetof(S, b); }
EOF
cp "$R/treeA/main.cpp" "$R/treeB/main.cpp"

export SCCACHE_DIR="$R/cache" SCCACHE_BASEDIRS="$R/treeA:$R/treeB" SCCACHE_CACHE_SIZE=1G
sccache --stop-server >/dev/null 2>&1 || true
sccache --start-server >/dev/null 2>&1

sccache g++ -c "$R/treeA/main.cpp" -I"$R/treeA/inc" -o "$R/a.o"
sccache g++ -c "$R/treeB/main.cpp" -I"$R/treeB/inc" -o "$R/b.o"

cmp -s "$R/a.o" "$R/b.o" && echo "MISCOMPILE: treeB got treeA's object"
objdump -d --no-show-raw-insn "$R/b.o" | grep -m1 -o '\$0x[0-9a-f]*'  # $0x4, want $0x8

Observed:

The last row is what localises it: with preprocessor cache mode off the content-based object key is computed and is correct, so the object cache itself is fine.

What to change - the manifest is not basedir-safe

src/compiler/c.rs, remember_include_file around line 1080:

let mut path = decode_path(path).context("failed to decode path")?;
if path.is_relative() {
    path = cwd.join(path);
}

That absolute path becomes the key of included_files, is carried into PreprocessorCacheEntry::add_result (preprocessor_cache.rs:93) and is what result_matches opens.

Two ways out, pick one and say why in the PR:

(a) Make the recorded paths tree-relative. Store an include that lives under a basedir stripped of it, and re-root it at validation time against this compilation's tree - the longest basedir that is a prefix of the current cwd. Paths outside every basedir (/usr/include/...) stay absolute and keep working as they do now. result_matches needs that root passed in, so it grows a parameter, and IncludeEntry needs to distinguish a stripped path from an absolute one - do not rely on is_relative() alone, a preprocessor can emit a relative path for a file that is not under any basedir. Bump FORMAT_VERSION (preprocessor_cache.rs:45), existing entries cannot be interpreted under the new rule.

(b) Refuse preprocessor cache mode when basedirs are set. Return None from preprocessor_cache_entry_hash_key when basedirs is non-empty, log once at info why, document the interaction in docs/Local.md and docs/Caching.md and next to SCCACHE_BASEDIRS in README.md. Correct, cheap, and it costs every basedirs user the direct-mode speedup.

(a) is the real fix; (b) is defensible as a stopgap if (a) turns out not to fit in one change. Do not ship both.

Either way the two features must stop being silently combinable, because today the failure is a wrong object file and not a miss.

Tests

Per AGENTS.md section 1. At minimum:

• Unit, mod test in src/compiler/preprocessor_cache.rs: two entries whose includes live in different trees under different basedirs must not validate against each other. Build the trees with tempfile, as the existing test_preprocessor_cache_entry_hash_key_basedirs does.
• End-to-end, tests/system.rs: the reproduction above, reduced. Compile the same source from two trees that differ only in a header, with both as basedirs, and assert the second is a miss and its object differs. This is the test that would actually have caught the bug, and there is currently nothing in the suite that compiles the same source from two basedirs and checks what came back rather than the hit count.

Run what CI runs (AGENTS.md section 4) and say which tests needed a real compiler and whether you ran them.

Docs

docs/Caching.md describes what goes into the hash and does not mention that a preprocessor cache hit bypasses it. Fix that while you are here - it is the sentence that would have made this bug obvious.

Also update README.md and docs/Configuration.md next to SCCACHE_BASEDIRS, and docs/Local.md for preprocessor cache mode, to state how the two interact after your change.

State of upstream (checked 2026-09-16)

upstream/main is 0f9467c. Against v0.18.0 its entire net diff is one deleted line in README.md. The direct-mode series that sits in the log - 85c5ecf .. 18dd633, "Add cache hit variant for direct mode" and friends - was reverted in full by c9d3943 .. 0f9467c, and was stats reporting only: it touched src/server.rs and src/compiler/compiler.rs, never the hash key or include validation. Nothing upstream fixes this.

Open work that is adjacent but does not touch include validation: #2711 (an earlier take on #2840), #2673 (-f*-prefix-map normalisation), #2814 (scoping basedirs to compiler requests). Confirmed by diffing each against remember_include_file / result_matches / IncludeEntry; #2571 and #2607 do show up there, but only as context lines from moving the preprocessor cache between modules.

Worth reading before settling on a design: issue #2595 (basedirs for git worktrees, the workflow this bug lives in) and #2766, the direct mode RFC.

Reporting it

There is no upstream issue for this. File one before or with the PR, in your own words (AGENTS.md section 2), and say plainly that it is a wrong-code bug in a released version so that users on 0.18.0 with basedirs know to clear $SCCACHE_DIR/preprocessor and turn direct mode off until it is fixed.