Bazel fails for cache deadline exceeded and cache unavailable errors

### Description of the bug:

Bazel seems to be unable to recover if inputs to a given action cannot be downloaded from cache, while it recovers in other cases.

Bazel fails with either:
- `Failed to fetch blobs because of a remote cache error.: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED`
- `Failed to fetch blobs because of a remote cache error.: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception`

Tested with a few versions of bazel (7.1.1, 7.4.0, last_green)
On `last_green` version bazel actually retries actions possibly due to `--experimental_remote_cache_eviction_retries` flag, but always fails if above issues re-appear.

Included examples where bazel always fails.
Note that none of those options helps in the above example: `--remote_local_fallback`, `--experimental_remote_cache_eviction_retries`.

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

`MODULE.bazel`:
```
module(name = "simple", version = "1.0")
```

`BUILD`:
```
genrule(
name = "first",
outs = ["first.txt"],
cmd = "echo -n first_content > \"$@\"",
)

genrule(
name = "second",
srcs = ["first.txt"],
outs = ["second.txt"],
cmd = "echo -n second_content > \"$@\"",
tags = ["no-cache"],
)
```

Patched version of remote-cache https://github.com/buchgr/bazel-remote that just sleeps on blob with "first_content":
```diff
$ git diff
diff --git a/server/grpc_bytestream.go b/server/grpc_bytestream.go
index b24969c..856f13b 100644
--- a/server/grpc_bytestream.go
+++ b/server/grpc_bytestream.go
@@ -7,6 +7,7 @@ import (
"io"
"strconv"
"strings"
+ "time"

"google.golang.org/genproto/googleapis/bytestream"
"google.golang.org/grpc/codes"
@@ -53,6 +54,12 @@ func (s *grpcServer) Read(req *bytestream.ReadRequest,
return err
}

+ if hash == "b8219593f9d14091f88892f0861f4e3af53c4db510950d32fae459241fc83017" {
+ s.accessLogger.Printf("---------- SLEEP START --------------- %s", req.ResourceName)
+ time.Sleep(2 * time.Minute)
+ s.accessLogger.Printf("---------- SLEEP STOP --------------- %s", req.ResourceName)
+ }
+
if size == 0 {
if cmp == casblob.Identity {
s.accessLogger.Printf("GRPC BYTESTREAM READ COMPLETED %s", req.ResourceName)
```

Running cache:
```bash
bazel run //:bazel-remote -- --dir /tmp/bazel-cache --max_size 10000 --grpc_address 127.0.0.1:9092
```

Running bazel:
```bash
bazel clean && bazel build :second --remote_cache="grpc://127.0.0.1:9092" --remote_timeout=1s
```

On second execution of bazel:
```
ERROR: /home/hauser/work/bazel_remote_cache_fail_issues/BUILD:7:8: Executing genrule //:second failed: Failed to fetch blobs because of a remote cache error.: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 0.999691193s. [closed=[], open=[[buffered_nanos=74747, remote_addr=127.0.0.1/127.0.0.1:9092]]]
```

For cache that fails during blob retireval, using this patch of bazel-remote:
```diff
$ git diff
diff --git a/server/grpc_bytestream.go b/server/grpc_bytestream.go
index b24969c..feaf024 100644
--- a/server/grpc_bytestream.go
+++ b/server/grpc_bytestream.go
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"io"
+ "os"
"strconv"
"strings"

@@ -53,6 +54,10 @@ func (s *grpcServer) Read(req *bytestream.ReadRequest,
return err
}

+ if hash == "b8219593f9d14091f88892f0861f4e3af53c4db510950d32fae459241fc83017" {
+ os.Exit(42)
+ }
+
if size == 0 {
if cmp == casblob.Identity {
s.accessLogger.Printf("GRPC BYTESTREAM READ COMPLETED %s", req.ResourceName)
```

Running bazel:
```
ERROR: /home/hauser/work/bazel_remote_cache_fail_issues/BUILD:7:8: Executing genrule //:second failed: Failed to fetch blobs because of a remote cache error.: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
```

### Which operating system are you running Bazel on?

Ubuntu 20.04

### What is the output of `bazel info release`?

release 7.4.0

### Have you found anything relevant by searching the web?

[No local fallback after cache timeout](https://github.com/bazelbuild/bazel/issues/20123) - This is possibly related issue, possibly this one is duplicate (although error messages are slightly different)
[Rethink spawn strategies](https://github.com/bazelbuild/bazel/issues/19904) - A bag of ideas how caching can change going forward