bazelbuild / bazelbuild/bazel

Bazel hangs when `--execution_log_compact_file` and `--build_event_binary_file` both point to named pipes

Open
#28,800 3 comments 0 reactions 0 assignees View on GitHub
help wanted P3 team-Local-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

**1. `SpawnLogModule.buildComplete()` blocks the Guava EventBus thread.**

`SpawnLogModule` is a subscriber to `BuildCompleteEvent`. Its handler calls `spawnLogContext.close()` synchronously. For `CompactSpawnLogContext`, `close()` blocks until `AsynchronousMessageOutputStream`'s background writer thread drains all pending entries through `ZstdOutputStream` and into the pipe. While the EventBus thread is blocked here, `BuildEventStreamer` — which is the *next* subscriber in the same dispatch cycle — never runs, so `BuildToolLogs` is never published to BEP and `CLOSE_EVENT_FUTURE` is never enqueued. The BEP writer thread stalls indefinitely waiting for events that will never arrive.

**2. `ByteStreamBuildEventArtifactUploader` reads the execlog pipe to compute a checksum.**

After `addLocalFile(outputPath)` is called, `ByteStreamBuildEventArtifactUploader.readPathMetadata()` calls `digestUtil.compute(path)` which reads the entire file to hash it. For a named pipe, this call blocks until the write-end is closed. But the write-end is held by the EventBus thread, which is stuck in cause #1. Full deadlock.

```
EventBus thread: SpawnLogModule.buildComplete()
└─ spawnLogContext.close() ──────── BLOCKED (pipe backpressure)
∴ BuildEventStreamer.buildComplete() never fires
∴ BuildToolLogs event never enqueued
∴ BEP writer stalls

ByteStream thread: readPathMetadata(execlog.pipe)
└─ digestUtil.compute(pipe) ─────── BLOCKED (write-end still open)
```

### Which category does this issue belong to?

_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

```bash
mkfifo /tmp/bep.pipe /tmp/execlog.pipe
cat /tmp/bep.pipe > /dev/null &
cat /tmp/execlog.pipe > /dev/null &
bazel build //some:target \
--build_event_binary_file=/tmp/bep.pipe \
--execution_log_compact_file=/tmp/execlog.pipe
# hangs forever; "Waiting for build events upload: BinaryFormatFileTransport"
```

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with SpawnLogModule.buildComplete(), BuildEventStreamer.buildComplete(), and ByteStreamBuildEventArtifactUploader.readPathMetadata(), then reproduce the hang using the mkfifo and bazel build commands in the issue. Trace the EventBus and BEP writer interactions; done means the build completes and both named-pipe outputs can be processed without the described deadlock.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.