bazelbuild / bazelbuild/bazel

Consider deduplicating digests for hardlinked/symlinked outputs

Open
#24,365 5 comments 3 reactions 0 assignees View on GitHub
P2 team-Performance type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 16h
Merged PRs (30d)
72

Description

Currently, when an action copies inputs to outputs by hardlinking or symlinking them, we recalculate digests for the output files, which can take a significant performance toll on builds that copy lots of files around. This could in theory be avoided if Bazel was able to recognize that it has seen the target of the link before (and hasn't been modified since).

For hardlinks, this could be as simple as using `(st_dev, st_ino, st_mtime, st_ctime, st_size)` as the `DigestUtils` cache key (instead of the current `(path, st_ino, st_mtime, st_ctime, st_size)`). Symlinks would require a bit more work, but are still doable. We'd have to think very carefully about the correctness implications, though.

cc @woody77 @fangism

Contributor guide

Open the contributing guide

Research direction

Start by inspecting DigestUtils and how its current path-based cache key is used when outputs are hardlinked or symlinked. Determine how link targets and file metadata can be recognized without reusing an outdated digest, with correctness for both link types as the completion criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.