googleapis / googleapis/google-cloud-go
spanner: data race in request_id_header.go during concurrent retries
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
While running tests with a custom gRPC interceptor for fault injection, a data race was detected in the Spanner client during concurrent retries.
### Client
Spanner
### Environment
- `go version go1.24.6 darwin/arm64`
- `cloud.google.com/go/spanner v1.86.0`
- `google.golang.org/grpc v1.76.0`
### Actual behavior
A data race is reported by `go test -race` under the following conditions:
- Multiple goroutines execute Spanner operations concurrently through a shared client
- Retries occur due to transient errors
**Race detector output:**
```
==================
WARNING: DATA RACE
Write at 0x00c000b8c1b0 by goroutine 6059:
google.golang.org/grpc.HeaderCallOption.after()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/rpc_util.go:226 +0x55
google.golang.org/grpc.(*clientStream).finish()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:1050 +0x36b
google.golang.org/grpc.newClientStreamWithParams.func4()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:402 +0x114
Previous write at 0x00c000b8c1b0 by goroutine 6075:
google.golang.org/grpc.HeaderCallOption.after()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/rpc_util.go:226 +0x55
google.golang.org/grpc.(*clientStream).finish()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:1050 +0x36b
google.golang.org/grpc.newClientStreamWithParams.func4()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:402 +0x114
Goroutine 6059 (running) created at:
google.golang.org/grpc.newClientStreamWithParams()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:397 +0x1d44
google.golang.org/grpc.newClientStream.func3()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:223 +0x1a4
google.golang.org/grpc.newClientStream()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/stream.go:258 +0xc87
cloud.google.com/go/spanner.(*requestIDHeaderInjector).interceptStream()
/home/runner/go/pkg/mod/cloud.google.com/go/spanner@v1.85.1/request_id_header.go:204 +0x2b5
cloud.google.com/go/spanner.(*requestIDHeaderInjector).interceptStream-fm()
:1 +0xde
google.golang.org/grpc.getChainStreamer.func1.getChainStreamer.1()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.75.1/clientconn.go:520 +0x183
cloud.google.com/go/spanner.allClientOpts.addStreamNativeMetricsInterceptor.metricsStreamInterceptor.func5()
/home/runner/go/pkg/mod/cloud.google.com/go/spanner@v1.85.1/client.go:751 +0xac
google.golang.org/grpc.getChainStreamer.func1()
```
### Root cause
The race appears to originate in `spanner/request_id_header.go`, within the `retryerWithRequestID.Resolve()` method.
This method creates a single `metadata.MD` instance that is reused across retry attempts:
```go
func (wr *retryerWithRequestID) Resolve(cs *gax.CallSettings) {
nthRequest := wr.gsc.nextNthRequest()
attempt := uint32(1)
md := new(metadata.MD)
wr.generateAndInsertRequestID(md, nthRequest, attempt)
cs.GRPC = append(cs.GRPC, grpc.Header(md))
...
newRetryer := func() gax.Retryer {
return (wrapRetryFn)(func(err error) (pause time.Duration, shouldRetry bool) {
attempt++
wr.generateAndInsertRequestID(md, nthRequest, attempt) // reused across retries
return originalRetryer.Retry(err)
})
}
cs.Retry = newRetryer
}
```
During concurrent use, the same `*metadata.MD` instance can be accessed by multiple goroutines:
- `generateAndInsertRequestID()` modifies the shared `metadata.MD` via `md.Set()` during retries
- `grpc.HeaderCallOption.after()` writes response headers to the same `*metadata.MD` instance
Contributor guide
Assessment
This issue has not been assessed yet.