pingcap / pingcap/tidb

statistics: document and regression-test stats background task reclassification

Open
#71,371 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Track regression coverage and user-facing release-note clarification for the intentional statistics request classification change in #69452, backported to `release-8.5` by #70658. Related design: #69362; original feature tracking: #69472.

This issue tracks an expected behavior change and its upgrade implications. It does **not** report a confirmed resource-isolation or foreground-performance defect.

### Behavior to preserve and document

Before this change, ANALYZE scans and several other internal statistics operations used the `stats` request source. An existing `BACKGROUND=(TASK_TYPES='stats', ...)` configuration could therefore throttle both.

After this change, ANALYZE scans retain `stats`, while affected statistics loading and maintenance operations use `StatsForegroundPriority`. This intentionally keeps query-critical statistics work from being delayed by ANALYZE background throttling.

| Operation | Before #70658 | After #70658 |
| --- | --- | --- |
| ANALYZE scan requests | `internal_stats`; eligible for the configured stats background limiter | Same |
| Affected non-ANALYZE statistics requests, including LOCK/UNLOCK STATS | `internal_stats`; eligible for the configured stats background limiter | `internal_StatsForegroundPriority`; no longer match the stats background rule |

The stored resource-group settings are preserved during an upgrade, but the set of requests matching an existing `TASK_TYPES='stats'` setting changes. During TiDB rolling upgrades, old and new TiDB instances can consequently apply different classifications to the same statistics maintenance operation. The new classification remains after the upgrade completes.

This is consistent with the design. The public documentation describes `stats` as manually or automatically triggered statistics collection; it does not promise to cover all internal statistics work. Not matching this particular background rule does not mean bypassing every resource-control mechanism.

### Regression case to add

Use a dedicated test cluster representing the reachable TiDB rolling phase: PD and TiKV at the target revision, with old and new TiDB frontends. The initial regression boundary is TiDB `v8.5.8` (`8b857efa20363d50a8fa2ea7dd9809a85a61b115`) versus the #70658 candidate (`94b637928d864d98743c49bfc5c6aa5fb5159cbc`, the reviewed `v8.5.9-pre` build). Future target builds must record their exact identity and contain the change.

The small workload is repeated `LOCK STATS` / `UNLOCK STATS` on dedicated test tables, plus `ANALYZE TABLE` controls. This is a behavior-regression test: it should pass when the intentional old/new difference is present.

- [ ] Add automated regression coverage with directly identified old/new SQL endpoints and retained version/configuration/metric evidence.
- [ ] Verify old LOCK/UNLOCK requests use `internal_stats`, while new requests use `internal_StatsForegroundPriority`.
- [ ] Verify ANALYZE scan requests remain `internal_stats` through both versions.
- [ ] With an existing `TASK_TYPES='stats'` policy and an activated limiter, verify the expected limiter-accounting difference. Include a background-disabled control; use isolated measurements and reject missing or contaminated evidence rather than treating it as a pass.
- [ ] Restore configuration and remove the test schema even when an assertion fails. Keep latency diagnostic; do not use a fixed elapsed-time ratio as the pass criterion.
- [ ] Also cover preservation of a pre-existing background policy across the actual upgrade, including other configured task types and the utilization limit. A preassembled mixed-version topology alone does not establish this migration property.

A previous bounded experiment observed the expected request-source change and limiter-wait difference. Its negative controls and cleanup verification were incomplete, so it is not a completed regression run or proof of production performance impact. Increased foreground latency/resource contention would require a separate representative workload.

### Regression implementation

Draft implementation: [PingCAP-QE/endless#4003](https://github.com/PingCAP-QE/endless/pull/4003) (QE repository). It includes the mixed-state case, TCMS Case definition, runbook, and six passing helper tests. The suite compiles; the new case has **not yet been executed on a live mixed-version cluster**. Full-upgrade configuration preservation remains a separate follow-up above.

### Release notes and documentation

The release-note text in #69452 and #70658 already describes the fresh-cluster ANALYZE default and preservation of resource-group settings during upgrades. **Please expand the user-facing release note to explain the classification change**, including:

- affected non-ANALYZE internal statistics requests leaving `stats` background throttling;
- the distinction between preserving stored settings and preserving the requests matched by those settings;
- temporary old/new frontend differences during a rolling upgrade;
- ANALYZE scans remaining eligible for the stats background limiter.

Suggested wording:

> Separate query-critical internal statistics work from ANALYZE background throttling. ANALYZE scans continue to use the `stats` task type; affected statistics loading and maintenance requests use a separate foreground-priority source. Upgrades preserve existing resource-group settings, while these internal requests no longer match `TASK_TYPES='stats'`. Old and new TiDB instances can exhibit different classification behavior during a rolling upgrade.

- [ ] Include this explanation in the applicable release notes and resource-control documentation.

### References

- [Final request-source definitions and rationale](https://github.com/pingcap/tidb/blob/94b637928d864d98743c49bfc5c6aa5fb5159cbc/pkg/kv/option.go#L196-L204)
- [Design: separate query-critical statistics work from background control](https://github.com/pingcap/tidb/blob/5c9d41aebb347ba3c100e214be726922f0a52670/docs/design/2026-06-22-integrate-analyze-into-background-task-resource-control.md#L107-L113)
- [Upgrade configuration rationale](https://github.com/pingcap/tidb/pull/69452#discussion_r3518664034)
- [Existing public definition of the stats task type](https://github.com/pingcap/docs/blob/42da4252914248472710bc8f9d3bb0546015093e/tidb-resource-control-background-tasks.md#L25-L33)

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/option.go and the existing PingCAP-QE/endless#4003 draft, then read the referenced resource-control documentation and upgrade rationale. Run the named mixed-version regression case only on the specified old/new endpoints; done means verified request classifications, limiter controls, cleanup, upgrade preservation, and expanded release-note documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems, documentation, testing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.