Local source directory metadata changes bypass checksum notification, causing stale COPY hits and later misses

Start with vendor/github.com/tonistiigi/fsutil/diskwriter.go, especially the existing-directory branch in DiskWriter.HandleChange, then trace NotifyHashed through session/filesync/diffcopy.go and cache/contenthash/checksum.go. Run the supplied buildctl reproduction on a dedicated Linux OCI/overlayfs worker. Done means directory metadata changes notify checksum updates and regression coverage demonstrates both stale COPY invalidation and reuse after filtered-context interleaving.

Bug description

Changing the permissions of an existing directory in a local build context can update the synchronized source filesystem without updating BuildKit's cached directory-header checksum. This causes two observable problems:

1. A COPY can incorrectly remain cached immediately after a directory-permission change.
2. After another build selects a different subset of the same context, the original unchanged COPY can execute again under a different content-checksum key. The previous COPY snapshot can still exist, and the two COPY outputs can have identical file contents and permissions.

The failure reproduces with two tiny files, FROM scratch, serial builds, a private worker/cache, and GC disabled. No registry cache, external base image, application compilation, concurrent build, or cache pruning is needed.

Reproduction

Use a fresh dedicated Linux BuildKit daemon with the OCI/overlayfs worker and --oci-worker-gc=false. Optional --save-cache-debug is useful for examining the checksum graph. Run the following on a Linux buildctl client, adjusting addr for that dedicated daemon:

#!/bin/sh
set -eu

addr=unix:///run/buildkit/buildkitd.sock
p=$(mktemp -d)
mkdir -p "$p/context/common/sub" "$p/context/other" "$p/dockerfiles"
printf 'unchanged source\n' > "$p/context/common/sub/file.txt"
printf 'other source\n' > "$p/context/other/file.txt"
chmod 755 "$p/context" "$p/context/common" "$p/context/common/sub" \
  "$p/context/other" "$p/dockerfiles"
chmod 644 "$p/context/common/sub/file.txt" "$p/context/other/file.txt"

printf 'FROM scratch\nCOPY common /common\n' > "$p/dockerfiles/a"
printf 'FROM scratch\nCOPY other /other\n' > "$p/dockerfiles/b"
printf 'FROM scratch\nCOPY common /copied\n' > "$p/dockerfiles/a2"

build() {
  buildctl --addr "$addr" build --progress=plain \
    --frontend=dockerfile.v0 \
    --local context="$p/context" \
    --local dockerfile="$p/dockerfiles" \
    --opt filename="$1"
}

# Establish a correctly cached 0755 source tree.
build a                    # 1: executes
build a                    # 2: CACHED

# Change only directory metadata.
chmod 2755 "$p/context/common" "$p/context/common/sub"
build a                    # 3: unexpectedly CACHED

# Different followpaths remove common from the reusable source snapshot.
build b                    # 4: executes
build a                    # 5: executes, now using fresh 2755 directory hashes
build a                    # 6: CACHED

# Create an output with actual 0755 directories but stale 2755 source hashes.
chmod 755 "$p/context/common" "$p/context/common/sub"
build a2                   # 7: executes

# Do not modify the context or Dockerfile between 7 and 9.
build b                    # 8: CACHED
build a2                   # 9: unexpectedly executes again
build a2                   # 10: CACHED

# With stable directory metadata, the same interleaving hits cache.
build b                    # 11: CACHED
build a2                   # 12: CACHED
build b                    # 13: CACHED
build a2                   # 14: CACHED

buildctl --addr "$addr" du --verbose

The numbered sequence above was executed against an isolated v0.30.0 worker. The script presents that sequence with portable temporary paths; the original execution used a buildctl client inside the isolated Linux pod and captured raw JSON progress.

Expected behavior

• Step 3 should invalidate the COPY because the copied directory metadata changed.
• Steps 7 and 9 have identical source bytes, source metadata, Dockerfile and operation. Step 9 should reuse step 7's result.
• Changing only which context paths another build selects should not change a content checksum for an unchanged tree.

Observed evidence

• All numbered COPY execution/cache outcomes are as annotated above.
• A complete source tar checksum was identical before and after steps 7 and 9: c9ca3941c67d779ffbea01ba70cc56612965aad07e26ab709e73a402341ae13a for the original fixture archive.
• Steps 7 and 9 produced separate retained snapshots. Both contained /copied and /copied/sub with mode 0755, and /copied/sub/file.txt with mode 0644, uid/gid 0, and SHA-256 a2d5d0958be77d7cb1e4509712fe51527f2f09d35c3594f5b8b4fac775ace2c8.
• Native cache-debug data distinguishes directory header mode 1517 (octal 2755) from 493 (octal 0755). The exact header digests are:
  • 2755: sha256:0b7cfcefc809fe1b555bfc8cd55fb059dd0b0987e497f8f03777689aeaa63a80
  • 0755: sha256:1a2805521573cb10d73bcd51d16bfa84c8f461d68599740e35484b90b2ab1bab
• Stable-metadata interleaving in steps 11–14 hit cache throughout.

Source analysis

In the vendored fsutil DiskWriter.HandleChange existing-directory branch:

if oldFi != nil && fi.IsDir() && oldFi.IsDir() {
    if err := rewriteMetadata(destPath, statCopy); err != nil {
        return errors.Wrapf(err, "error setting dir metadata for %s", destPath)
    }
    return nil
}

rewriteMetadata changes the filesystem ownership/mode/timestamps, but this branch returns before processChange can hash the new directory metadata and invoke NotifyCb.

BuildKit's recvDiffCopy supplies CacheUpdater.HandleChange as NotifyHashed. fsutil's normal Receive path passes that to DiskWriterOpt.NotifyCb. The local-source handler reuses the existing contenthash context and saves it after synchronization; the existing-directory update above never tells it that the header changed. The checksum implementation can then return the existing digest without rescanning.

When filtered context reuse removes and later recreates a directory, the newly created directory takes the notifying path. The same actual source can therefore acquire a different checksum depending on which directory headers were retained from earlier synchronizations.

A likely fix is to ensure metadata updates to existing directories notify the checksum updater, with regression coverage for both stale hits and later duplicate misses. No patched BuildKit binary has been tested as part of this report.

Version information and scope

Runtime reproduction:

buildkitd github.com/moby/buildkit v0.30.0 dd2170e156c9633da1b2d1a58a6188e3f7d36fa4
buildctl  github.com/moby/buildkit v0.30.0 dd2170e156c9633da1b2d1a58a6188e3f7d36fa4

• Linux amd64, rootful OCI worker, overlayfs snapshotter.
• Image: moby/buildkit@sha256:57269d1784e49b46228c45a1a1b870fbe40e0a639ab60b37b032d83af5bccdfc.
• Fresh private cache, GC disabled, serial builds, no exporter specified.
• The minimal reproduction used ordinary local directories; it does not require NFS.

v0.33.0 was source-audited, not runtime-tested here. Its vendored diskwriter.go is byte-for-byte identical to the file in the reproduced v0.30.0 deployment. The relevant Receive/callback path is still used. cache/contenthash/checksum.go differs only by an unrelated errors.Errorf("not found") to errors.New("not found") change. The missing notification also remains in the inspected v0.31.2 and v0.32.2 vendored code. No version of introduction is claimed.

I searched the BuildKit and fsutil issue trackers and did not find an exact match. Related #3821 concerns changed source permissions despite a COPY --chmod override; #4817 concerns regular-file content changes with preserved metadata. This reproduction instead changes directory mode, skips its checksum notification, and then exhibits history-dependent hashes during filtered context reuse.