googleapis / googleapis/google-cloud-go

bigtable: potential memory leak from xDS runner goroutine when DirectPath is unavailable, introduced in v1.46.0

Open
#14,582 21 comments 0 reactions 1 assignee Claimed by @sushanb View on GitHub
api: bigtable triage me
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Client

v1.46.0+

## Environment

go 1.26.3

## Code and Dependencies

```go
// initialize bigtable client
bigtableClientClientConf := gcpBigtable.ClientConfig{
AppProfile: appProfile,
MetricsProvider: gcpBigtable.NoopMetricsProvider{},
}
bigtableClient, btErr := gcpBigtable.NewClientWithConfig(ctx, projectID, instanceID, bigtableClientClientConf, option.WithGRPCConnectionPool(cfg.BigtableConnectionPoolSize))
...
```

go.mod

```text
module modname

go 1.25.0

require (
cloud.google.com/go/bigtable v1.47.0
)
```

## Expected behavior

Memory usage is stable, consistent with v1.43.0–v1.45.0 behavior

## Actual behavior

Heap grows continuously after upgrading to v1.46.0. Heap profile shows `(*adsStreamImpl).runner` from `google.golang.org/grpc@v1.80.0/internal/xds/clients/xdsclient/ads_stream.go` accumulating memory. Which caused OOM in one of our pods.

## Screenshots
Memory Usage Trends
Image

Heap Live Size
Image

## Additional context
We are suspecting the bug is the unconditional DirectPath probe spawning xDS goroutines that can't clean up on non-DirectPath infrastructure.
The trigger is v1.46.0 enabling BigtableChannelPool by default, which exposed this to all users.

Our temporary workaround :
After setting `CBT_ENABLE_DIRECTPATH=false`, the memory usage returns to stable:
Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.