perf: reduce memory usage of bes-uploader
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
we've been able to improve the *overall* performance of our builds by improving the throughput of `bes-uploader`. when events are not processed quickly, the `eventQueue` and `ackQueue` may grow faster than they're cleared, meaning these queues consume more and more of the heap. sometimes this causes bazel to OOM, but in other cases the build is slowed down because of extra competition for memory.
we've seen two reasons for `bes-uploader` processing events slowly:
1. slow remote server. we've seen this when the CAS is slow, but I imagine a slow BES would have the same effect
2. `bes-uploader` does too much work, see https://github.com/bazelbuild/bazel/pull/20574 and https://github.com/bazelbuild/bazel/pull/20575
there are things we can do to address both of these, but it would be nice if the `bes-uploader` wasn't able to cause the rest of the build to perform poorly, even if it can't clear the events quickly.
note that `bes_upload_mode=fully-async` does not help because the events still need to be stored in memory.
some ideas:
1. reduce size of `SendRegularBuildCommand` by using `PathConverter` ASAP. if paths were converted when an event is pushed to the queue, then `PathConverter` instances could be collected immediately. some `PathConverter` instances for our monorepo are 22mb (see screenshot below)
2. serialize events before appending to the `eventQueue`
4. offload some of the queue to disk (maybe using mmap)
### Which category does this issue belong to?
Performance
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
_No response_
### Which operating system are you running Bazel on?
macos, linux
### What is the output of `bazel info release`?
6.4.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
_No response_
### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by locating the bes-uploader implementation and its eventQueue and ackQueue handling, then profile or reproduce behavior with a slow remote server. Define done as preventing queue growth from exhausting or materially contending for the build's heap, while preserving event delivery; the issue lists several possible designs but no specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100