HTTP remote cache: upload timeouts are never retried (`--remote_retries` does not apply), and RuntimeException-class upload failures are silently dropped with `--remote_cache_async` (default)

### Description of the bug

Two related defects in the HTTP remote-cache upload path (`--remote_cache=http(s)://...`),
found while investigating action-result (AC) entries that never reached a GCS
HTTP cache from a nightly CI pipeline. Both verified against the 8.6.0
sources; defect 1 is additionally reproduced live (transcript below).

#### Defect 1 — upload timeouts are never retried; the AC entry is lost (reproduced)

1. `HttpCacheClient` arms the upload pipeline with
`IdleTimeoutHandler(timeoutSeconds, WriteTimeoutException.INSTANCE)`
(`remote/http/HttpCacheClient.java:328`), where `timeoutSeconds` is
`--remote_timeout`. A PUT with no channel activity for that period
(e.g. stalled mid-write by uplink saturation while several large
outputs upload concurrently) fires it.
2. `HttpUploadHandler.exceptionCaught` translates `WriteTimeoutException`
into `UploadTimeoutException extends IOException`
(`remote/http/HttpUploadHandler.java:127-128`), message
`"Upload of '%s' timed out. Sent %d bytes."`.
3. `RETRIABLE_HTTP_ERRORS` (`remote/RemoteModule.java:197-227`) matches
`ClosedChannelException`, `DownloadTimeoutException`, `HttpException`
with 5xx status, `IOException`s whose message contains
`"connection reset"` or `"operation timed out"`, and one specific
`DecoderException` (a netty-bug workaround). `UploadTimeoutException`
matches none of these — note the asymmetry: the **download** timeout is
class-matched retriable, the **upload** timeout is not. `--remote_retries`
(default 5) therefore never applies to a timed-out upload.
4. The AC upload has no retry protection even independent of 3.:
`uploadBlob` (CAS) is wrapped in `retrier.executeAsync`
(`remote/http/HttpCacheClient.java:722`), but `uploadActionResult`
calls `uploadAsync` directly with no retrier
(`remote/http/HttpCacheClient.java:752-760`), and
`CombinedCache.uploadActionResult` adds none either — so classifying
the timeout as retriable would still leave the AC PUT single-attempt.
5. `UploadManifest.upload` deliberately uploads the ActionResult last, after
all CAS blobs (`remote/UploadManifest.java:614-626`). Correct ordering,
but combined with 3. a single stalled CAS PUT means the AC entry is never
written at all while sibling uploads from the same build survive — a
partial-loss pattern that is very confusing to attribute.

Observed effect (both with `--remote_cache_async` default `true` and with
`=false` — behavior is identical): every stalled PUT is attempted exactly
once, the AC entry is never PUT, the build exits 0 with `Build completed
successfully`, and the only diagnostic is an easily-missed aggregated
warning (printed at end of invocation with async uploads, at upload
completion with sync):

```
WARNING: Remote Cache: 3 errors during bulk transfer:
com.google.devtools.build.lib.remote.http.UploadTimeoutException: Upload of '/cas/3a4b58...' timed out. Sent 147 bytes.
com.google.devtools.build.lib.remote.http.UploadTimeoutException: Upload of '/cas/b30cff...' timed out. Sent 783 bytes.
com.google.devtools.build.lib.remote.http.UploadTimeoutException: Upload of '/cas/1328f3...' timed out. Sent 67108864 bytes.
```

When the action is hours long, the silently-lost AC entry is expensive:
every downstream consumer re-executes it, and nothing in the producer's
build output points at the cause.

#### Defect 2 — RuntimeException-class upload failures are completely silent on the async path (code-verified)

The upload machinery reports and (in principle) retries only
`IOException`-family failures. A `RuntimeException` failing an upload
escapes every handler on the async path and is dropped without a trace:

1. `AbstractHttpHandler.exceptionCaught` fails the upload promise with the
**raw** `Throwable` (`remote/http/AbstractHttpHandler.java:115-118`);
`HttpUploadHandler.exceptionCaught` translates only
`WriteTimeoutException` and `TooLongFrameException`, everything else
passes through raw (`remote/http/HttpUploadHandler.java:126-134`).
Raw RuntimeExceptions are also seeded outside netty, e.g.
`IllegalStateException("Channel pipeline is not empty.")` in
`acquireUploadChannel` (`remote/http/HttpCacheClient.java:322`) and the
`checkState` guards in `HttpUploadHandler` (`"handler can't be shared
between pipelines."`, `"response before request"`) — channel-pool
reuse invariant guards, each of which fails an upload with a raw
`RuntimeException`.
2. `Utils.getFromFuture` rethrows `RuntimeException` causes raw
(`remote/util/Utils.java:125-126`).
3. `Utils.waitForBulkTransfer` aggregates only `IOException` into
`BulkTransferException` (`remote/util/Utils.java:592-627`, per-transfer
catch at `:605`); a `RuntimeException` aborts the bulk wait raw and
abandons the remaining transfers.
4. The escaping `RuntimeException` bypasses `doUploadOutputs`'s
`catch (IOException e) { reportUploadError(e); }`, and with
`--remote_cache_async=true` (the default since Bazel 8) the background
upload task catches only `ExecException` and `InterruptedException`
(`remote/RemoteExecutionService.java:1874-1885`). The exception kills
the background thread: nothing reaches the `Reporter`, no warning is
printed, no retry happens, and the AC entry (uploaded last, see defect
1 item 5) is lost. On the synchronous path the same exception would
propagate out of the spawn runner and surface visibly; async makes it
invisible.

### Observed in practice

A nightly CI pipeline (Bazel 8.6.0,
`--remote_cache=https://storage.googleapis.com/`,
`--remote_timeout=900`, `--remote_retries=10`,
`--remote_upload_local_results=true`) executed actions with
multi-hundred-MB outputs on three machines uploading concurrently. A subset
of AC entries never reached the bucket — GCS soft-delete forensics confirm
the PUTs never landed (nothing deleted them) — interleaved minute-by-minute
with successful uploads from the same jobs, and with **zero** upload
warnings in any log. Two of the lost keys had been executed and "uploaded"
by two machines each; all four writes were lost. Consumers re-executed the
lost actions at digest-verified identical action keys hours later, one of
them a 40+ minute action.

The complete absence of the bulk-transfer warning rules out the (warned)
timeout flavor and matches defect 2's silent path. After switching the
pipeline to `--remote_cache_async=false` no further losses have been
observed.

### How to reproduce

Defect 1, reproduced with release 8.6.0. Stub cache that accepts a PUT and
then stalls (never reads the body, never responds):

```python
#!/usr/bin/env python3
# stub.py PORT — HTTP "cache" that 404s GETs and stalls PUTs forever.
import socket, sys, threading, time

def handle(conn):
buf = b""
while b"\r\n\r\n" not in buf:
chunk = conn.recv(4096)
if not chunk:
return
buf += chunk
method, path = buf.split(b"\r\n", 1)[0].decode().split(" ")[:2]
print(time.strftime("%H:%M:%S"), method, path, flush=True)
if method == "GET":
conn.sendall(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n")
conn.close()
elif method == "PUT":
while True: # stop reading; never respond
time.sleep(3600)

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 131072)
srv.bind(("127.0.0.1", int(sys.argv[1]))); srv.listen(64)
while True:
threading.Thread(target=handle, args=(srv.accept()[0],), daemon=True).start()
```

Workspace: `MODULE.bazel` with `module(name = "repro")` and

```python
genrule(
name = "big",
outs = ["big.bin"],
cmd = "head -c 67108864 /dev/urandom > $@",
)
```

Run `python3 stub.py 8917 &`, then:

```
bazel --ignore_all_rc_files build //:big \
--remote_cache=http://127.0.0.1:8917 \
--remote_timeout=5 \
--remote_upload_local_results=true
```

Observed (identical with and without `--remote_cache_async=false`, and with
`--remote_retries` at its default of 5):

- the build exits 0, `Build completed successfully`;
- the aggregated `WARNING: Remote Cache: 3 errors during bulk transfer:
...UploadTimeoutException...` shown above;
- the stub log shows each CAS PUT attempted exactly **once** — no retry
despite `--remote_retries=5` — and **no PUT to `/ac/...` at all**:

```
10:36:42 GET /ac/3a4b5864... -> 404
10:36:43 PUT /cas/3a4b5864... (stalls)
10:36:43 PUT /cas/1328f395... (stalls)
10:36:43 PUT /cas/b30cff0f... (stalls)
(no further requests; no /ac/ PUT ever)
```

Defect 2 is a code-path analysis (exact citations above); its real-world
signature is the "Observed in practice" section: uploads lost with zero
warnings, which the warned timeout path cannot produce.

### Suggested fixes (any of, ideally several)

- Catch `Throwable` (or at least `RuntimeException`) in the async
background upload task in `RemoteExecutionService.uploadOutputs` and
route it through `reportUploadError`, so async upload failures are at
least visible.
- Classify `UploadTimeoutException` as retriable in
`RETRIABLE_HTTP_ERRORS`, symmetric with `DownloadTimeoutException`,
and wrap `HttpCacheClient.uploadActionResult` in the retrier like
`uploadBlob` so AC uploads get retry protection at all.
- Aggregate `RuntimeException` transfer failures in
`Utils.waitForBulkTransfer` (or wrap `RuntimeException` causes into
`IOException` at the upload-path `getFromFuture` boundary) so the
existing warn machinery applies.

### Which category does this issue belong to?

Remote Execution

### Which operating system are you running Bazel on?

Linux (x86_64)

### What is the output of `bazel info release`?

release 8.6.0

### Related issues

- #25484 (`bazel clean` stuck, likely due to async HTTP uploads — same
async upload subsystem)
- #27179 (the RxJava → virtual-threads rework that shaped the current async
upload path; the catch clauses analyzed above are from that
implementation)
- #21578 (async upload made default)
- #15279, #14008 (earlier async-upload-path correctness bugs)
- #23250 (BEP reports files as uploaded when the upload failed)
- #23079 (earlier proposal to classify more remote-cache errors as
retriable)

Start with the stalled-PUT reproduction, then read remote/http/HttpCacheClient.java, HttpUploadHandler.java, RemoteModule.java, Utils.java, and RemoteExecutionService.java at the cited paths. Trace both timeout and RuntimeException failures through synchronous and asynchronous upload paths. Done means the reported failure modes are retried where intended and async failures are surfaced or aggregated instead of being silently lost.

Written by the indexing model from the issue text.