Consider deduplicating digests for hardlinked/symlinked outputs
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 72
Description
Currently, when an action copies inputs to outputs by hardlinking or symlinking them, we recalculate digests for the output files, which can take a significant performance toll on builds that copy lots of files around. This could in theory be avoided if Bazel was able to recognize that it has seen the target of the link before (and hasn't been modified since).
For hardlinks, this could be as simple as using `(st_dev, st_ino, st_mtime, st_ctime, st_size)` as the `DigestUtils` cache key (instead of the current `(path, st_ino, st_mtime, st_ctime, st_size)`). Symlinks would require a bit more work, but are still doable. We'd have to think very carefully about the correctness implications, though.
cc @woody77 @fangism
Contributor guide
Research direction
Start by inspecting DigestUtils and how its current path-based cache key is used when outputs are hardlinked or symlinked. Determine how link targets and file metadata can be recognized without reusing an outdated digest, with correctness for both link types as the completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100