apache / apache/doris

[Enhancement] Introduce unified sensitive-field redaction for FE SQL logs

Open
#65,875 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I searched the existing open and closed issues and found no similar issue.

### Description

FE diagnostic logs are produced by multiple SQL processing paths, and these paths do not currently share one consistent policy for handling sensitive property values.

Removing the complete SQL text from logs makes production troubleshooting significantly harder. The desired behavior is to preserve the SQL structure, identifiers, and other non-sensitive content while redacting only values associated with confirmed sensitive fields, such as passwords, access keys, secret keys, tokens, SASL credentials, and equivalent connector-specific properties.

This issue tracks a general FE logging improvement and hardening effort. It is not intended to describe or track a specific vulnerability.

### Proposed solution

Introduce a shared SQL log redaction facility with the following properties:

1. Preserve non-sensitive SQL content and statement structure so that logs remain useful for diagnosis.
2. Replace only the values of known sensitive fields. Reuse a central sensitive-key registry where possible and support statement-specific keys for resources, stages, routine load, catalogs, repositories, and similar property-bearing statements.
3. Provide consistent handling for:
- successfully parsed statements;
- parser-error paths;
- SQL dialect conversion paths;
- statements whose parser or plan object is unavailable.
4. Use a fail-safe fallback: if the redactor cannot safely process a statement, do not emit the original SQL from that path. Log metadata such as query ID, statement type, length, user, dialect, and error type for that event instead.
5. Migrate the main FE diagnostic logging entry points to the shared facility, including:
- StmtExecutor and ConnectProcessor;
- HTTP statement execution endpoints;
- parser and dialect-conversion logging;
- Load command logging;
- CBO/rewrite logging;
- query profile logging.
6. Keep the regular diagnostic-log policy aligned with, but clearly separated from, audit-log encryption behavior.

### Tests

Add log-capture tests that verify:

- known sensitive values never appear in emitted logs;
- SQL structure and non-sensitive content remain available;
- statement-specific sensitive keys are handled;
- parser failures and dialect-conversion failures use the safe fallback;
- all migrated entry points apply the same policy.

### Are you willing to submit PR?

- [ ] Yes, I am willing to submit a PR.

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct.

Contributor guide

Open the contributing guide

Research direction

Start by mapping the listed FE diagnostic entry points, including StmtExecutor, ConnectProcessor, HTTP statement execution, parser and dialect-conversion logging, Load commands, CBO/rewrite logging, and query profiles. Then inspect the existing sensitive-key registry and the separation between diagnostic logging and audit-log encryption. Done means log-capture tests show sensitive values are absent, useful SQL content is preserved, failures use safe metadata fallback, and all listed paths share the policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, observability, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.