tikv / tikv/pd

[release-8.5] Existing loopback metric-storage starts returning HTTP 502 after upgrade

Open
#11,149 1 comment 0 reactions 0 assignees View on GitHub
contribution type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Configure every PD member on v8.5.7 with an explicit PD-local Prometheus endpoint:

```toml
[pd-server]
metric-storage = "http://127.0.0.1:19090"
```

The endpoint was healthy in each PD Pod's network namespace. Query `/pd/api/v1/metric/query`, then perform a normal PD rolling upgrade without changing the configuration and query through the new PD leader.

This only affects users who explicitly configured `metric-storage` to loopback, for example because Prometheus or a query proxy runs as a sidecar, host-local agent, or node-local service. If `metric-storage` was never configured (the default is empty), this issue has no effect.

### What did you expect to see?

An unchanged configuration that worked on v8.5.7 should either keep working after a patch upgrade, or be rejected before the upgrade/configuration is applied with a clear migration error.

### What did you see instead?

- The v8.5.7 leader returned HTTP 200 with valid Prometheus JSON, and the local responder's request counter increased.
- The current release-8.5 leader returned HTTP 502:

```json
{"status":"error","errorType":"proxy","error":"metric query failed"}
```

The target responder stayed healthy but received no request, proving that PD rejected the loopback target before dialing it. As a control, changing only the address to a routable service DNS name returned HTTP 200 on both versions.

The security hardening in #11107 intentionally rejects loopback/private special destinations for the metric proxy, but PD still accepts and stores the old loopback configuration. Therefore the incompatibility is discovered only when Dashboard or an API client queries metrics. During a mixed PD phase, success can depend on the serving member; after all PD members are upgraded, the PD metric proxy consistently returns 502 until the address is migrated.

The user-visible scope is metric-backed Dashboard panels and callers of `/pd/api/v1/metric/query` or `/pd/api/v1/metric/query_range`. SQL traffic, TiKV availability, and data correctness are not affected.

### What version of PD are you using (`pd-server -V`)?

- Baseline: PD v8.5.7, Git `6dce4a68e3e93b523657bd05cba89a86d64de117`
- Affected/current release-8.5: Git `2417ccf64a77be7576b3aad890ec98399e2522bf` (`v8.5.8-pre`)

### Compatibility addendum

- Introducing backport: #11107 (`server: harden metric query proxy`)
- Prerequisite: an explicitly configured loopback `metric-storage`; default/unset configurations are unaffected.
- Runtime evidence: http://3.107.106.196/dashboard/validations/a7220632-75ef-43df-8834-5fded698a993

Contributor guide

Open the contributing guide

Research direction

Start by reading the metric query proxy hardening in #11107 and tracing how the configured metric-storage value reaches /pd/api/v1/metric/query and /pd/api/v1/metric/query_range. Compare behavior for loopback and routable addresses across the v8.5.7 and current release-8.5 versions; done means the old configuration remains usable or produces a clear migration error before queries return HTTP 502.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.