bazelbuild / bazelbuild/bazel

Computed C++ linker RAM is way off reality

Open
#17,368 23 comments 5 reactions 1 assignee Claimed by @bigelephant29 View on GitHub
P2 team-Local-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

In our C++ builds, we have observed Bazel eagerly running more than a few link actions at once. Each of these linker executions can easily consume 8GB, so combining just a handful together can easily exceed RAM limits. Using `-c dbg` bumps the memory consumption to 20GB per linker, which is ridiculous. We are currently using ld.

The puzzling thing was that even using something like `--local_ram_resources` to limit Bazel to a few GB, say 2GB, did not help at all. Bazel kept scheduling too many linkers at once. This made me think that the RAM computations for the link actions were not working correctly.

Upon looking I noticed commit 01c10e030c1, which mentions that the new limits were set based on data collected at Google. Thus I'm surprised that this new behavior didn't work well.

I ended up patching `CppLinkAction.java` to dump the resources computed in `estimateResourceConsumptionLocal` and found that all of our link actions are assumed to only need 50MBs, which is way off reality. I also noticed that our problematic actions only have about ~600 inputs.

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Come up with a link action that, with a small number of inputs, consumes multiple GBs. (This is what we observe but I have no idea at this point why the linker is behaving in this way.)

Craft a build file that contains multiple independent such actions.

Run the build and see Bazel trigger too many linkers at once, overwhelming the machine.

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

release 6.0.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

There is https://github.com/bazelbuild/bazel/issues/6477 which would help if we had it, but I would prefer to avoid crafting our own rules just to paper over a deficiency of existing native rules. A tag or other attribute on the existing rules would work better from a usability perspective.

### Any other information, logs, or outputs that you want to share?

I do not think the C++ link rule can provide a local resource set that works in all cases, if only because there are different linkers and they have different memory needs, and because even the same linker can exhibit vastly different behavior depending on compilation modes (see 8GB vs. 20GB when enabling dbg mode).

There should be a way for downstream users to configure the resource sets of the C++ rules without having to modify the Bazel source code. I'd suggest supporting `cpu:X` and `ram:X` tags or similar that override whatever the rules compute; this would have been sufficient for us to workaround the problem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.