bazelbuild / bazelbuild/bazel

Move I/O out of `--experimental_throttle_remote_action_building` semaphore

Open
#28,734 3 comments 0 reactions 1 assignee Claimed by @fmeum View on GitHub
P2 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

Bazel currently relies on `--experimental_throttle_remote_action_building` to throttle the creation of new Merkle trees. This serves two unrelated purposes:

1) Creating Merkle trees is CPU-intensive and rarely uses I/O (only for source directories, which are cached). It makes sense to not start building too many trees in parallel as a remote action can start only when its Merkle tree is ready.
2) Merkle trees can be large and keeping too many around in memory at the same time risks OOMs. The flag goes as far as limiting the number of trees retained in memory at the same time by including the upload of a tree in the span of the semaphore.

Many users have reported that this flag hurts their RBE throughput in unacceptable ways. This is likely due to 2), as building of new trees is delayed until previous trees have been uploaded. Thus we should try to reduce the retained size of Merkle trees as much as possible and move the upload out of the semaphore.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.