googleapis / googleapis/google-cloud-go
storage: gRPC Reader.Close() unconditionally cancels stream context, causing otelgrpc to report successful ReadObject spans as errors
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
## Client
Storage (gRPC client, `storage.NewGRPCClient`)
## Environment
Linux / GKE, Go 1.23+. Reproduced against `cloud.google.com/go/storage` v1.50.0 through v1.62.1 — behavior identical across all tested versions.
## Code and Dependencies
```go
package main
import (
"context"
"io"
"cloud.google.com/go/storage"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"go.opentelemetry.io/otel"
"google.golang.org/api/option"
"google.golang.org/grpc"
)
func main() {
ctx := context.Background()
tp := newTracerProvider() // any OTel tracer provider
otel.SetTracerProvider(tp)
client, _ := storage.NewGRPCClient(ctx,
option.WithGRPCDialOption(grpc.WithStatsHandler(otelgrpc.NewClientHandler())),
)
defer client.Close()
r, _ := client.Bucket("b").Object("o").NewReader(ctx)
_, _ = io.Copy(io.Discard, r) // fully drain, returns cleanly
_ = r.Close() // span closes with status=Error, "context canceled"
}
```
go.mod
```text
module repro
go 1.23
require (
cloud.google.com/go/storage v1.62.1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.58.0
go.opentelemetry.io/otel v1.33.0
google.golang.org/api v0.215.0
google.golang.org/grpc v1.69.2
)
```
## Expected behavior
A successful `ReadObject` (all expected bytes received, no transport error) should produce a gRPC client span with `status.code = OK`.
## Actual behavior
Every `google.storage.v2.Storage/ReadObject` span is reported with `status.code = Error` and `status.message = "context canceled"`, even though `io.Copy` completed cleanly and all bytes were received.
**Root cause:** `storage/grpc_reader.go:403-410` unconditionally calls the per-stream `cancel()` in `Close()`:
```go
func (r *gRPCReadObjectReader) Close() error {
if r.cancel != nil {
r.cancel()
}
r.stream = nil
r.currMsg = nil
return nil
}
```
Because `cancel()` fires even when the reader has already delivered all requested bytes, the per-stream context is cancelled on an otherwise-successful RPC. otelgrpc then records the cancellation as the span's terminal status, overwriting what should have been `OK`.
## Screenshots
N/A — reproducible deterministically with any OTel span exporter (stdout, OTLP, Jaeger, etc.).
## Additional context
### Why this matters
- Every object read appears as a failed RPC; dashboards, SLOs, and alerts built on gRPC span status become unusable or need per-call exception lists.
- Noise masks real cancellations — operators cannot distinguish a genuine client-abort from a benign `Close`.
- No caller-side workaround exists without wrapping/shimming the returned `*Reader`, which is impractical since `Reader` is a concrete type.
Contributor guide
Assessment
This issue has not been assessed yet.