bazelbuild / bazelbuild/bazel

BEP does not report memory usage of test actions

Open
#30,953 0 comments 1 reaction 0 assignees View on GitHub
team-Core type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 16h
Merged PRs (30d)
72

Description

### Description of the problem

Bazel measures the peak memory usage of locally executed spawns but does not expose it
in the Build Event Protocol, so BEP/BES consumers cannot attribute memory usage to a
test target without post-processing a separate execution log.

The measurement already exists. `process-wrapper` and `linux-sandbox` write `getrusage(2)`
(including `maxrss`) into a `stats.out` proto, and Bazel reads it back in
`LocalSpawnRunner` and `AbstractSandboxSpawnRunner` via
`SpawnResult.Builder#setResourceUsageFromProto`, landing in
`SpawnMetrics.measuredMemoryPeakBytes`.

The BEP field for it also already exists — `TestResult.ExecutionInfo.ResourceUsage`
(`name`/`value`) in `build_event_stream.proto` — but `addResourceUsage` is never called
anywhere in the tree. The only consumer of the measurement today is the execution log, as
`SpawnExec.metrics.measured_memory_peak_bytes`.

### Feature requests: what underlying problem are you trying to solve with this feature?

Tracking per-test memory usage over time from the BES stream, to catch tests whose memory
footprint regresses and to size test execution resources. The execution log can answer this
offline, but it is a separate artifact that has to be joined back to test targets, which
makes it awkward for a BES-based pipeline that already receives `TestResult` events.

### Design questions

Three things I would like maintainer input on before this is worth polishing:

1. **Is `resource_usage` intentionally unpopulated in Bazel?** The field exists in the proto
but is dead in this tree, which suggests it may be filled in internally with an
established set of key names. If so, matching those names and units is preferable to
inventing new ones. Bytes with a unit-bearing name (`memory_bytes`) is consistent with
`measured_memory_peak_bytes` in the execution log; `"memory"` in KB would instead match
`SpawnResult#getMemoryInKb`.

2. **Cache hits report nothing.** `TestResult#getCachedTestAttempts` passes
`ExecutionInfo.getDefaultInstance()`, so a locally cached test carries no memory figure.
Surviving a cache hit requires a new field on `TestResultData` in `test_status.proto`,
since that is what gets serialized into the test's cache status artifact. For a CI
pipeline in steady state most test targets are cache hits, so without this the data is
sparse — but it is also a proto change with a wider blast radius.

3. **`maxrss` semantics.** As noted at `SpawnResult.java:805`, this is the largest single
child process rather than the peak of the process tree, so it understates tests that
fork. `LinuxSandboxedSpawnRunner` already creates a `VirtualCgroup` per spawn, so reading
`memory.peak` would be more accurate, but that is a larger change and covers only the
Linux sandbox. Is reporting the `maxrss` number acceptable given it is already what the
execution log reports?

### Coverage

Measured today, so reportable: `local` (with process-wrapper), `linux-sandbox`,
`darwin-sandbox`.

Not measured, so omitted rather than reported as zero: remote execution and remote cache
hits (REAPI's `ExecutedActionMetadata` has no memory field), Windows
(`WindowsSandboxedSpawn#getStatisticsPath` returns `null`), and local cache hits (question 2).

### What operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

Applies to Bazel at HEAD.

### Have you found anything relevant by searching the web?

`--experimental_collect_local_action_metrics` and
`--experimental_collect_local_sandbox_action_metrics` were moved to the options graveyard in
549254e48c and are no-ops; the statistics they used to gate are now collected unconditionally
whenever a spawn runs under the process-wrapper or a reporting sandbox.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing measurement from process-wrapper and linux-sandbox through LocalSpawnRunner, AbstractSandboxSpawnRunner, SpawnResult.Builder#setResourceUsageFromProto, and SpawnMetrics.measuredMemoryPeakBytes. Then inspect build_event_stream.proto, TestResult#getCachedTestAttempts, and test_status.proto to determine the agreed resource_usage and cache behavior. Done means measured local and sandbox test actions expose the memory value in BEP while unsupported and cache-hit cases remain omitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
api, build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.