googleapis / googleapis/google-cloud-go

spanner: data race in request_id_header.go during concurrent retries

Open
#13,223 0 comments 0 reactions 1 assignee Claimed by @rahul2393 View on GitHub
api: spanner triage me
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

While running tests with a custom gRPC interceptor for fault injection, a data race was detected in the Spanner client during concurrent retries.

### Client

Spanner

### Environment

- `go version go1.24.6 darwin/arm64`
- `cloud.google.com/go/spanner v1.86.0`
- `google.golang.org/grpc v1.76.0`

### Actual behavior

A data race is reported by `go test -race` under the following conditions:

- Multiple goroutines execute Spanner operations concurrently through a shared client
- Retries occur due to transient errors

**Race detector output:**

```
==================
WARNING: DATA RACE
Write at 0x00c000b8c1b0 by goroutine 6059:
google.golang.org/grpc.HeaderCallOption.after()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/rpc_util.go:226 +0x55
google.golang.org/grpc.(*clientStream).finish()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:1050 +0x36b
google.golang.org/grpc.newClientStreamWithParams.func4()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:402 +0x114
Previous write at 0x00c000b8c1b0 by goroutine 6075:
google.golang.org/grpc.HeaderCallOption.after()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/rpc_util.go:226 +0x55
google.golang.org/grpc.(*clientStream).finish()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:1050 +0x36b
google.golang.org/grpc.newClientStreamWithParams.func4()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:402 +0x114
Goroutine 6059 (running) created at:
google.golang.org/grpc.newClientStreamWithParams()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:397 +0x1d44
google.golang.org/grpc.newClientStream.func3()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:223 +0x1a4
google.golang.org/grpc.newClientStream()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:258 +0xc87
cloud.google.com/go/spanner.(*requestIDHeaderInjector).interceptStream()
/home/runner/go/pkg/mod/cloud.google.com/go/spanner@v1.85.1/request_id_header.go:204 +0x2b5
cloud.google.com/go/spanner.(*requestIDHeaderInjector).interceptStream-fm()
:1 +0xde
google.golang.org/grpc.getChainStreamer.func1.getChainStreamer.1()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/clientconn.go:520 +0x183
cloud.google.com/go/spanner.allClientOpts.addStreamNativeMetricsInterceptor.metricsStreamInterceptor.func5()
/home/runner/go/pkg/mod/cloud.google.com/go/spanner@v1.85.1/client.go:751 +0xac
google.golang.org/grpc.getChainStreamer.func1()
```

### Root cause

The race appears to originate in `spanner/request_id_header.go`, within the `retryerWithRequestID.Resolve()` method.
This method creates a single `metadata.MD` instance that is reused across retry attempts:

```go
func (wr *retryerWithRequestID) Resolve(cs *gax.CallSettings) {
nthRequest := wr.gsc.nextNthRequest()
attempt := uint32(1)
md := new(metadata.MD)
wr.generateAndInsertRequestID(md, nthRequest, attempt)
cs.GRPC = append(cs.GRPC, grpc.Header(md))
...
newRetryer := func() gax.Retryer {
return (wrapRetryFn)(func(err error) (pause time.Duration, shouldRetry bool) {
attempt++
wr.generateAndInsertRequestID(md, nthRequest, attempt) // reused across retries
return originalRetryer.Retry(err)
})
}
cs.Retry = newRetryer
}
```

During concurrent use, the same `*metadata.MD` instance can be accessed by multiple goroutines:

- `generateAndInsertRequestID()` modifies the shared `metadata.MD` via `md.Set()` during retries
- `grpc.HeaderCallOption.after()` writes response headers to the same `*metadata.MD` instance

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.