bazelbuild / bazelbuild/bazel

Hermetic Linux sandbox reports test.log modified during flaky timeout retries

Open
#31,064 0 comments 0 reactions 0 assignees View on GitHub
category: sandboxing team-Local-Exec type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

With the hermetic Linux sandbox enabled, a timed-out test with multiple flaky attempts is converted from a normal test timeout (exit 3) into an environmental error (exit 36).

Bazel's fallback XML-generation spawn consumes `test.log`, and the failed-test attempt lifecycle then moves/reuses that path.

### Which category does this issue belong to?

Hermetic Linux sandbox

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Use this reproducer: https://gist.github.com/Spikhalskiy/e63308a1ce39ffeab2a9a055fad63912

Run `./repro.sh`. It uses the official Bazel 9.2.0 Linux image and reproduces reliably, exiting 36.

Controls:

- Removing `--experimental_use_hermetic_linux_sandbox` reports `TIMEOUT in 3 out of 3` and exits 3.
- Keeping hermetic mode but setting `--flaky_test_attempts=1` reports a normal timeout and exits 3.

### Which operating system are you running Bazel on?

Linux amd64 (official Bazel container)

### What is the output of `bazel info release`?

release 9.2.0

### Have you found anything relevant by searching the web?

This looks related to #14008, but the reproducer does not use a remote cache or `--remote_cache_async`.

### Any other information, logs, or outputs that you want to share?

Observed output:

```text
Testing //:timeout_test failed: I/O exception during sandboxed execution:
input dependency .../testlogs/timeout_test/test.log was modified during execution.
//:timeout_test NO STATUS
Executed 0 out of 1 test: 1 was skipped.
```

Contributor guide

Open the contributing guide

Research direction

Start by running the linked reproducer's ./repro.sh with Bazel 9.2.0 and compare hermetic mode with --flaky_test_attempts=1. Trace the timeout_test attempt handling around test.log and fallback XML generation. Done means flaky timeout retries remain a normal timeout with exit 3 instead of reporting a modified test.log and exit 36.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
build-system, operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.