bazelbuild / bazelbuild/bazel

perf: reduce memory usage of bes-uploader

Open
#20,579 3 comments 0 reactions 0 assignees View on GitHub
P2 team-Performance type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

we've been able to improve the *overall* performance of our builds by improving the throughput of `bes-uploader`. when events are not processed quickly, the `eventQueue` and `ackQueue` may grow faster than they're cleared, meaning these queues consume more and more of the heap. sometimes this causes bazel to OOM, but in other cases the build is slowed down because of extra competition for memory.

we've seen two reasons for `bes-uploader` processing events slowly:
1. slow remote server. we've seen this when the CAS is slow, but I imagine a slow BES would have the same effect
2. `bes-uploader` does too much work, see https://github.com/bazelbuild/bazel/pull/20574 and https://github.com/bazelbuild/bazel/pull/20575

there are things we can do to address both of these, but it would be nice if the `bes-uploader` wasn't able to cause the rest of the build to perform poorly, even if it can't clear the events quickly.

note that `bes_upload_mode=fully-async` does not help because the events still need to be stored in memory.

some ideas:
1. reduce size of `SendRegularBuildCommand` by using `PathConverter` ASAP. if paths were converted when an event is pushed to the queue, then `PathConverter` instances could be collected immediately. some `PathConverter` instances for our monorepo are 22mb (see screenshot below)
2. serialize events before appending to the `eventQueue`
4. offload some of the queue to disk (maybe using mmap)

Screenshot 2023-12-18 at 12 54 54 pm

### Which category does this issue belong to?

Performance

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

macos, linux

### What is the output of `bazel info release`?

6.4.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the bes-uploader implementation and its eventQueue and ackQueue handling, then profile or reproduce behavior with a slow remote server. Define done as preventing queue growth from exhausting or materially contending for the build's heap, while preserving event delivery; the issue lists several possible designs but no specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.