googleapis / googleapis/google-cloud-go

storage: gRPC Reader.Close() unconditionally cancels stream context, causing otelgrpc to report successful ReadObject spans as errors

Open
#14,470 5 comments 0 reactions 1 assignee Claimed by @cpriti-os View on GitHub
api: storage triage me
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Client

Storage (gRPC client, `storage.NewGRPCClient`)

## Environment

Linux / GKE, Go 1.23+. Reproduced against `cloud.google.com/go/storage` v1.50.0 through v1.62.1 — behavior identical across all tested versions.

## Code and Dependencies

```go
package main

import (
"context"
"io"

"cloud.google.com/go/storage"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"go.opentelemetry.io/otel"
"google.golang.org/api/option"
"google.golang.org/grpc"
)

func main() {
ctx := context.Background()

tp := newTracerProvider() // any OTel tracer provider
otel.SetTracerProvider(tp)

client, _ := storage.NewGRPCClient(ctx,
option.WithGRPCDialOption(grpc.WithStatsHandler(otelgrpc.NewClientHandler())),
)
defer client.Close()

r, _ := client.Bucket("b").Object("o").NewReader(ctx)
_, _ = io.Copy(io.Discard, r) // fully drain, returns cleanly
_ = r.Close() // span closes with status=Error, "context canceled"
}
```


go.mod

```text
module repro

go 1.23

require (
cloud.google.com/go/storage v1.62.1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.58.0
go.opentelemetry.io/otel v1.33.0
google.golang.org/api v0.215.0
google.golang.org/grpc v1.69.2
)
```

## Expected behavior

A successful `ReadObject` (all expected bytes received, no transport error) should produce a gRPC client span with `status.code = OK`.

## Actual behavior

Every `google.storage.v2.Storage/ReadObject` span is reported with `status.code = Error` and `status.message = "context canceled"`, even though `io.Copy` completed cleanly and all bytes were received.

**Root cause:** `storage/grpc_reader.go:403-410` unconditionally calls the per-stream `cancel()` in `Close()`:

```go
func (r *gRPCReadObjectReader) Close() error {
if r.cancel != nil {
r.cancel()
}
r.stream = nil
r.currMsg = nil
return nil
}
```
Because `cancel()` fires even when the reader has already delivered all requested bytes, the per-stream context is cancelled on an otherwise-successful RPC. otelgrpc then records the cancellation as the span's terminal status, overwriting what should have been `OK`.

## Screenshots

N/A — reproducible deterministically with any OTel span exporter (stdout, OTLP, Jaeger, etc.).

## Additional context

### Why this matters

- Every object read appears as a failed RPC; dashboards, SLOs, and alerts built on gRPC span status become unusable or need per-call exception lists.
- Noise masks real cancellations — operators cannot distinguish a genuine client-abort from a benign `Close`.
- No caller-side workaround exists without wrapping/shimming the returned `*Reader`, which is impractical since `Reader` is a concrete type.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.