grafana / grafana/pyroscope

Helm single-binary v2: default query-backend.address (dns:///_grpc._tcp...) is unresolvable by grpc-go → reads hang 30s / HTTP 499

Open
#5,229 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

### Describe the bug

**Chart:** pyroscope 2.0.2 · **Image:** grafana/pyroscope:2.0.2 · **Mode:** single-binary, `architecture.storage.v2: true`, `architecture.microservices.enabled: false`

In single-binary v2 mode, profiles ingest and compact fine, but **every profile-data read query hangs ~30s and returns HTTP 499**, so flame graphs render empty in Grafana.

Affected RPCs: `/querier.v1.QuerierService/SelectMergeStacktraces` and `/querier.v1.QuerierService/SelectSeries`.
Unaffected (return in ~1ms): `ProfileTypes`, `Series`, `LabelNames`, `LabelValues`, `GetProfileStats`.

The `query-backend` component logs **nothing** during the hang — the request never reaches it.

### Root cause

For single-binary v2, `operations/pyroscope/helm/pyroscope/templates/deployments-statefulsets.yaml` renders:

```
-query-backend.address=dns:///_grpc._tcp.-headless.$(NAMESPACE_FQDN):9095
```

The query-backend client (`pkg/querybackend/client/client.go`) hands this straight to `grpc.NewClient(address, ...)` with the stock grpc-go `dns` resolver and a service config of `waitForReady: true`.

grpc-go's `dns` resolver (v1.81.0) does an **A/AAAA lookup on the literal host**. It only does SRV for grpclb (`_grpclb._tcp.`), and `EnableSRVLookups` is `false` by default:

```go
// google.golang.org/grpc/internal/resolver/dns/dns_resolver.go
EnableSRVLookups = false
func (d *dnsResolver) lookupSRV(...) { ... d.resolver.LookupSRV(ctx, "grpclb", "tcp", d.host) }
func (d *dnsResolver) lookupHost(...) { addrs, err := d.resolver.LookupHost(ctx, d.host) }
```

The host `_grpc._tcp.` has an **SRV** record but **no A/AAAA record**, so the resolver yields zero endpoints. With `waitForReady: true`, every call parks until the 30s call timeout → HTTP 499. The metadata RPCs are unaffected because the metastore client uses its own `kubernetes://` discovery (`pkg/metastore/client`), not grpc-go's resolver — which is why `-metastore.address=kubernetes:///:9095` works while the query-backend address does not.

### To Reproduce

1. Deploy the Helm chart in single-binary mode with `architecture.storage.v2: true`.
2. Push any profile (ingestion + compaction succeed).
3. Open a flame graph in Grafana Explore (or call `SelectMergeStacktraces`).
4. It hangs ~30s, then returns 499; `query-backend` logs nothing.

### Expected behavior

`query-backend.address` resolves to the in-process query-backend gRPC endpoint and reads return promptly.

### Workaround

Override to a plain headless A record (grpc-go-resolvable), mirroring the working `metastore.address` form:

```yaml
pyroscope:
extraArgs:
query-backend.address: "dns:///pyroscope-headless.$(NAMESPACE_FQDN):9095"
```

(The chart skips its broken default when the key is present in `extraArgs`, so there is no duplicate flag.)

### Suggested fix

Drop the `_grpc._tcp.` SRV prefix from the chart default — use `dns:///-headless.$(NAMESPACE_FQDN):9095` (plain A records). Note `kubernetes://` / `dnssrvnoa+` are not options for this address because it is handed directly to grpc-go's resolver, which only understands `dns`/`passthrough` unless a custom resolver is registered.

### Environment

- Pyroscope: 2.0.2 (single-binary, v2 storage)
- Helm chart: pyroscope-2.0.2
- grpc-go: v1.81.0
- Kubernetes: GKE

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.