pingcap / pingcap/tidb

executor: collect producer-side evidence for precise statement RU accounting

Open
#70,747 1 comment 0 reactions 0 assignees View on GitHub
component/executor sig/planner type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Statement RU accounting can now traverse the whole physical-plan forest (Main, deduplicated CTE definitions, and scalar-subquery trees), but its calibration remains `Incomplete`. The current value is deliberately neither an exact value nor a mathematical upper/lower bound because several formula inputs have already lost execution-lifecycle or response-level information by the time the statement-RU consumer runs.

This issue tracks the producer-side evidence needed to tighten that contract without inferring execution from missing rows or multiplying shared CTE work by consumer count.

### Missing evidence

1. **Physical scan bytes per response/request.**
- Cloud TiKV can provide total key size for scan and point-get responses.
- TiDB should consume that response-level total directly (including close-time unconsumed responses) and aggregate it under the request root.
- Until then, estimating bytes from merged `ScanDetail` ratios can bias either upward or downward when requests have different key-size distributions.

2. **Formula-ready Sort/TopN work per executor lifecycle.**
- Repeated CTE rounds and correlated Apply executions merge row counts by plan ID.
- `sum(n_i * log(n_i))` cannot be reconstructed from `sum(n_i) * log(sum(n_i))`; the latter overestimates repeated work.
- Sort/TopN producers should publish a bounded sufficient statistic plus begun/completed/incomplete coverage, rather than an unbounded vector of row counts.

3. **Direct recursive-CTE and Apply execution-opportunity state.**
- Preserve three distinct states: intentionally skipped, executed with zero rows, and required evidence missing.
- CTE evidence should be keyed by `IDForStorage` and classify seed/recursive phase attempts, legal skip reasons, and completed rounds.
- Apply evidence should classify outer rows, filters, cache hits, logical inner invocations, and physical inner executions/cancellation.

### Ownership and lifetime

The direct executors should publish display-neutral, value-only snapshots into the statement-owned runtime evidence collector before live CTE/Apply objects are closed. Synchronization stays private to the provider/container; copied snapshots contain only scalar values and enums. Plan ID remains an execution-evidence lookup key, not an occurrence identity.

The statement-RU consumer should continue to:

- count each CTE producer once through the `IDForStorage`-deduplicated definition forest;
- keep skipped, executed-zero, and missing evidence distinct;
- fail closed on invalid or structurally inconsistent evidence;
- avoid treating positive aggregate rows or EOF as proof that every physical opportunity was attempted.

### Acceptance criteria

- Response-level scan/point-get total key size is consumed without reconstructing it from merged ratios.
- Repeated Sort/TopN work uses producer-accumulated per-lifecycle sufficient statistics.
- Recursive CTE and serial/parallel Apply expose complete opportunity/disposition snapshots.
- Regression tests cover registered-zero versus missing, empty/limited recursion, Apply filtering/cache/early close, repeated nonlinear work, and close-time unconsumed responses.
- Statement RU can advertise a tighter calibration only when all required producer evidence is complete; partial evidence must remain `Incomplete`.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the statement-RU consumer and the statement-owned runtime evidence collector, then inspect the direct scan, Sort/TopN, recursive-CTE, and Apply executors that publish evidence. Use the acceptance criteria as the completion checklist: complete response, lifecycle, and execution-opportunity snapshots must tighten calibration, while partial or inconsistent evidence remains Incomplete; add regression coverage for the listed edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.