[release-8.5] Existing loopback metric-storage starts returning HTTP 502 after upgrade
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Configure every PD member on v8.5.7 with an explicit PD-local Prometheus endpoint:
```toml
[pd-server]
metric-storage = "http://127.0.0.1:19090"
```
The endpoint was healthy in each PD Pod's network namespace. Query `/pd/api/v1/metric/query`, then perform a normal PD rolling upgrade without changing the configuration and query through the new PD leader.
This only affects users who explicitly configured `metric-storage` to loopback, for example because Prometheus or a query proxy runs as a sidecar, host-local agent, or node-local service. If `metric-storage` was never configured (the default is empty), this issue has no effect.
### What did you expect to see?
An unchanged configuration that worked on v8.5.7 should either keep working after a patch upgrade, or be rejected before the upgrade/configuration is applied with a clear migration error.
### What did you see instead?
- The v8.5.7 leader returned HTTP 200 with valid Prometheus JSON, and the local responder's request counter increased.
- The current release-8.5 leader returned HTTP 502:
```json
{"status":"error","errorType":"proxy","error":"metric query failed"}
```
The target responder stayed healthy but received no request, proving that PD rejected the loopback target before dialing it. As a control, changing only the address to a routable service DNS name returned HTTP 200 on both versions.
The security hardening in #11107 intentionally rejects loopback/private special destinations for the metric proxy, but PD still accepts and stores the old loopback configuration. Therefore the incompatibility is discovered only when Dashboard or an API client queries metrics. During a mixed PD phase, success can depend on the serving member; after all PD members are upgraded, the PD metric proxy consistently returns 502 until the address is migrated.
The user-visible scope is metric-backed Dashboard panels and callers of `/pd/api/v1/metric/query` or `/pd/api/v1/metric/query_range`. SQL traffic, TiKV availability, and data correctness are not affected.
### What version of PD are you using (`pd-server -V`)?
- Baseline: PD v8.5.7, Git `6dce4a68e3e93b523657bd05cba89a86d64de117`
- Affected/current release-8.5: Git `2417ccf64a77be7576b3aad890ec98399e2522bf` (`v8.5.8-pre`)
### Compatibility addendum
- Introducing backport: #11107 (`server: harden metric query proxy`)
- Prerequisite: an explicitly configured loopback `metric-storage`; default/unset configurations are unaffected.
- Runtime evidence: http://3.107.106.196/dashboard/validations/a7220632-75ef-43df-8834-5fded698a993
Contributor guide
Research direction
Start by reading the metric query proxy hardening in #11107 and tracing how the configured metric-storage value reaches /pd/api/v1/metric/query and /pd/api/v1/metric/query_range. Compare behavior for loopback and routable addresses across the v8.5.7 and current release-8.5 versions; done means the old configuration remains usable or produces a clear migration error before queries return HTTP 502.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100