pingcap / pingcap/tidb

Retain original statement summary samples when persistence is enabled and redact file output

Open
#70,965 0 comments 0 reactions 0 assignees View on GitHub
component/observability type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**
Statement summary currently applies `tidb_redact_log` when collecting `QUERY_SAMPLE_TEXT`. Enabling log redaction therefore also removes or marks the in-memory sample shown by statement-summary queries and TiDB Dashboard, even when the operator only wants to control persisted logs. The background writer also preserves the mode from collection instead of applying the setting in effect when it writes the file.

Observed in the source at TiDB master `5acf6574288567bb473762e42313e25880c02419`: `ExecStmt.GetOriginalSQL` formats using the session mode, while statement-log serialization writes `StmtRecord.SampleSQL` unchanged.

**Describe the feature you'd like:**
When the existing read-only instance setting `tidb_stmt_summary_enable_persistent` is ON, keep the original statement sample in memory (retaining sensitive-statement protections and sample-length limits), and apply the current GLOBAL `tidb_redact_log` when serializing the sample into the persistent statement log. When persistence is OFF, retain the existing collection-time redaction behavior. Do not add a separate timing variable. Persistent output modes are:

- OFF: original sample.
- ON: the already captured normalized SQL, without literal values or prepared arguments.
- MARKER: the sample with the existing redaction marker escaping.

Apply the same behavior to window rotation, shutdown/flush, and evicted-record logging without modifying records held by memory readers. For example, a sample captured with OFF and persisted after switching to ON should remain original in memory and be normalized in the file.

**Describe alternatives you've considered:**
Always keeping collection-time redaction prevents retrieving original samples in persistent deployments. Always retaining raw samples would change non-persistent deployments. A separate timing variable adds an unnecessary configuration choice when the existing persistence setting can select the behavior. Reparsing SQL while writing is unreliable because samples can be truncated or include a prepared-argument suffix; reuse the existing normalized SQL instead.

**Teachability, Documentation, Adoption, Migration Strategy:**
This changes in-memory `QUERY_SAMPLE_TEXT` only when persistence is enabled. Persistence remains disabled by default, preserving existing behavior for non-persistent deployments. The existing instance setting is read-only from SQL. History tables combine memory and persisted rows and can therefore contain both original and redacted samples. Access privileges stay unchanged. Existing log files are not rewritten. MARKER is reversible marking, not removal of literal values. This request does not change PLAN, PREV_SAMPLE_TEXT, or slow-log formatting.

Contributor guide

Open the contributing guide

Research direction

Start at ExecStmt.GetOriginalSQL and the statement-log serialization that writes StmtRecord.SampleSQL, then trace the persistent writer through window rotation, shutdown/flush, and evicted-record logging. Verify the existing sensitive-statement and sample-length protections and reuse the captured normalized SQL rather than reparsing. Done means memory readers retain the specified sample while persistent output follows the current GLOBAL tidb_redact_log mode in every write path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.