resource_control: fallback keyspace_name labels can leak in metrics cleanup
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
While addressing Resource Control observability in https://github.com/tikv/pd/pull/10867, we found a metric cleanup risk around `keyspace_name` labels.
Some Resource Control metric caches are keyed by `keyspaceID` / resource group / RU type, but Prometheus children are created and deleted by label values, including `keyspace_name`. If a metric child is created while the keyspace name cache misses and the code falls back to `keyspace-`, a later cleanup may resolve the real keyspace name from storage and call `DeleteLabelValues` with that real name instead of the fallback label used at creation time.
One possible sequence:
1. A Resource Control metric is created for keyspace ID `123` while the keyspace name is not cached.
2. The metric child is created with `keyspace_name="keyspace-123"`.
3. Later, the keyspace name cache is populated with the real name, for example `tenant-a`.
4. Cleanup calls `DeleteLabelValues` with `keyspace_name="tenant-a"`.
5. The Prometheus child created with `keyspace_name="keyspace-123"` is not deleted.
PR https://github.com/tikv/pd/pull/10867 fixed this for the newly added request metrics by:
- avoiding storage lookup in the non-background token request hot path, and
- storing the label values used to create request metric children, then deleting those exact label values during label rotation or cleanup.
The older Resource Control metric caches should still be audited and fixed with the same principle.
### What did you expect to see?
Metric cleanup should delete the exact Prometheus children that were created, even if a keyspace label changes from a fallback name such as `keyspace-123` to the real keyspace name later.
The request path should also avoid storage lookups only for metric label resolution.
### What did you see instead?
Fallback `keyspace_name` metric children can remain after cleanup if the cleanup path resolves a different keyspace name than the one used when the child was created.
This can leave stale Prometheus series and create duplicate or polluted Grafana curves, for example one series under `keyspace-123` and another under the real keyspace name. The impact is observability cardinality/resource leakage rather than Resource Control correctness.
### What version of PD are you using (`pd-server -V`)?
Found while working on the `release-nextgen-202603` Resource Control observability backport. The same pattern should be checked on `master` if the metric cache/cleanup code is shared.
Contributor guide
Assessment
This issue has not been assessed yet.