bazelbuild / bazelbuild/bazel

[7.4.1] Bazel clean is stuck, likely due to async uploads to HTTP remote cache

Open
#25,484 4 comments 0 reactions 1 assignee Claimed by @coeuvre View on GitHub
P2 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

We're experiencing intermittent hangs of `bazel clean` for 60+ minutes in a project utilizing an HTTP remote cache. The project is configured with `--experimental_remote_cache_async`, `--remote_timeout=60s`, Bazel 7.4.1, and BWOB. During these hangs, stack traces captured via `Ctrl+\` consistently show multiple threads blocked in `com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:797)`.

This suggests that `bazel clean` is waiting for asynchronous remote cache uploads to complete before deleting the output directory. While it's understandable to delay cleaning until all output artifacts are uploaded, it appears these HTTP cache uploads can become indefinitely stuck, bypassing the `--remote_timeout` setting. Consequently, `bazel clean` remains unresponsive for extended periods.

Interestingly, executing `bazel clean --remote_cache=` resolves the issue, completing the clean operation within seconds. This behavior seems counterintuitive.

I am not sure if this is by design or a bug. If it's by design, I'd like to suggest to change it. At least for me it would feel more natural to kill all unfinished async remote cache uploads once `bazel clean` is invoked by the user.

### Which category does this issue belong to?

Core

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I was not able to reproduce this with an Open Source project and I can not share our internal code base where this behavior can be observed quite frequently.

In our internal code base, especially on Mac ARM and Windows, this sometimes triggers the bug:

```
bazel build //...
bazel clean # will hang
Ctrl+c
bazel clean --remote_cache= # will finish within seconds
```

### Which operating system are you running Bazel on?

Mac (ARM)

### What is the output of `bazel info release`?

release 7.4.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

Slack thread: https://bazelbuild.slack.com/archives/CA31HN1T3/p1741269451254749

### Any other information, logs, or outputs that you want to share?

Here are snippets from the stack traces:

```
"Common-Cleaner" #18 [27395] daemon prio=8 os_prio=31 cpu=12.92ms elapsed=2637.84s tid=0x000000012983e200 nid=27395 waiting on condition [0x0000000171826000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x0000000302115f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.5/LockSupport.java:269)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.5/AbstractQueuedSynchronizer.java:1852)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.5/ReferenceQueue.java:71)
at java.lang.ref.ReferenceQueue.remove0(java.base@21.0.5/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.5/ReferenceQueue.java:218)
at jdk.internal.ref.CleanerImpl.run(java.base@21.0.5/CleanerImpl.java:140)
at java.lang.Thread.runWith(java.base@21.0.5/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.5/Thread.java:1583)
at jdk.internal.misc.InnocuousThread.run(java.base@21.0.5/InnocuousThread.java:186)

"block-waiting-0" #646 [67347] prio=5 os_prio=31 cpu=0.12ms elapsed=140.69s tid=0x000000012815ae00 nid=67347 in Object.wait() [0x00000009030b9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait0(java.base@21.0.5/Native Method)
- waiting on <0x0000000361273060> (a io.netty.util.concurrent.DefaultPromise)
at java.lang.Object.wait(java.base@21.0.5/Object.java:366)
at java.lang.Object.wait(java.base@21.0.5/Object.java:339)
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:254)
- locked <0x0000000361273060> (a io.netty.util.concurrent.DefaultPromise)
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:35)
at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:451)
at com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:797)
- locked <0x000000030acef978> (a java.lang.Object)
at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.close(DiskAndRemoteCacheClient.java:75)
at com.google.devtools.build.lib.remote.RemoteCache.deallocate(RemoteCache.java:498)
at io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at com.google.devtools.build.lib.remote.RemoteActionContextProvider.afterCommand(RemoteActionContextProvider.java:265)
at com.google.devtools.build.lib.remote.RemoteModule.afterCommandTask(RemoteModule.java:972)
at com.google.devtools.build.lib.remote.RemoteModule.lambda$afterCommand$5(RemoteModule.java:947)
at com.google.devtools.build.lib.remote.RemoteModule$$Lambda/0x00000070018eb448.call(Unknown Source)
at com.google.devtools.build.lib.runtime.BlockWaitingModule.lambda$submit$0(BlockWaitingModule.java:73)
at com.google.devtools.build.lib.runtime.BlockWaitingModule$$Lambda/0x00000070018eb668.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@21.0.5/Executors.java:572)
at java.util.concurrent.FutureTask.run(java.base@21.0.5/FutureTask.java:317)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.5/ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.5/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.5/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.5/Thread.java:1583)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.