executor: collect producer-side evidence for precise statement RU accounting
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Statement RU accounting can now traverse the whole physical-plan forest (Main, deduplicated CTE definitions, and scalar-subquery trees), but its calibration remains `Incomplete`. The current value is deliberately neither an exact value nor a mathematical upper/lower bound because several formula inputs have already lost execution-lifecycle or response-level information by the time the statement-RU consumer runs.
This issue tracks the producer-side evidence needed to tighten that contract without inferring execution from missing rows or multiplying shared CTE work by consumer count.
### Missing evidence
1. **Physical scan bytes per response/request.**
- Cloud TiKV can provide total key size for scan and point-get responses.
- TiDB should consume that response-level total directly (including close-time unconsumed responses) and aggregate it under the request root.
- Until then, estimating bytes from merged `ScanDetail` ratios can bias either upward or downward when requests have different key-size distributions.
2. **Formula-ready Sort/TopN work per executor lifecycle.**
- Repeated CTE rounds and correlated Apply executions merge row counts by plan ID.
- `sum(n_i * log(n_i))` cannot be reconstructed from `sum(n_i) * log(sum(n_i))`; the latter overestimates repeated work.
- Sort/TopN producers should publish a bounded sufficient statistic plus begun/completed/incomplete coverage, rather than an unbounded vector of row counts.
3. **Direct recursive-CTE and Apply execution-opportunity state.**
- Preserve three distinct states: intentionally skipped, executed with zero rows, and required evidence missing.
- CTE evidence should be keyed by `IDForStorage` and classify seed/recursive phase attempts, legal skip reasons, and completed rounds.
- Apply evidence should classify outer rows, filters, cache hits, logical inner invocations, and physical inner executions/cancellation.
### Ownership and lifetime
The direct executors should publish display-neutral, value-only snapshots into the statement-owned runtime evidence collector before live CTE/Apply objects are closed. Synchronization stays private to the provider/container; copied snapshots contain only scalar values and enums. Plan ID remains an execution-evidence lookup key, not an occurrence identity.
The statement-RU consumer should continue to:
- count each CTE producer once through the `IDForStorage`-deduplicated definition forest;
- keep skipped, executed-zero, and missing evidence distinct;
- fail closed on invalid or structurally inconsistent evidence;
- avoid treating positive aggregate rows or EOF as proof that every physical opportunity was attempted.
### Acceptance criteria
- Response-level scan/point-get total key size is consumed without reconstructing it from merged ratios.
- Repeated Sort/TopN work uses producer-accumulated per-lifecycle sufficient statistics.
- Recursive CTE and serial/parallel Apply expose complete opportunity/disposition snapshots.
- Regression tests cover registered-zero versus missing, empty/limited recursion, Apply filtering/cache/early close, repeated nonlinear work, and close-time unconsumed responses.
- Statement RU can advertise a tighter calibration only when all required producer evidence is complete; partial evidence must remain `Incomplete`.
Contributor guide
Research direction
Start by tracing the statement-RU consumer and the statement-owned runtime evidence collector, then inspect the direct scan, Sort/TopN, recursive-CTE, and Apply executors that publish evidence. Use the acceptance criteria as the completion checklist: complete response, lifecycle, and execution-opportunity snapshots must tighten calibration, while partial or inconsistent evidence remains Incomplete; add regression coverage for the listed edge cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100