Privilege DCL reports success while etcd publication fails — remote nodes keep stale authentication/authorization
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. Deploy a 2-node TiDB cluster.
2. Warm the privilege cache on the remote node (e.g. log in with the current credential).
3. Inject a transient etcd-write failure on the `/etcdserverpb.KV/Put` service for the `/tidb/privilege` key (~10s window; a gRPC fault proxy targeting only that service works — a TCP cut on the PD port kills TSO so the DCL cannot commit).
4. Run a DCL (`ALTER USER` / `GRANT` / `REVOKE`) on the other node during the window.
### 2. What did you expect to see? (Required)
Either the DCL fails, or the publication is retried until success; remote nodes converge to the new authentication/authorization.
### 3. What did you see instead (Required)
The DCL returns success to the client while the etcd publication failed (`etcd-cli put kv failed [key=/tidb/privilege]` retryCnt=0..4 then dropped, `notify update privilege failed`). The prewarmed remote node keeps:
- accepting the old password for new logins (stale authentication), and
- executing statements that were just revoked (stale authorization),
until the next successful publication converges (bounded ≤10 min window).
Root cause: privilege DCL commit and cross-node publication are non-atomic — the commit succeeds locally, but the etcd publication failure is not surfaced to the client, so remote nodes operate on a stale privilege handle snapshot.
Production reachability: rare (fault-common × DCL-low-frequency conjunction; the etcd-write fault itself is routine — PD maintenance/upgrade/disk-latency windows happen regularly). Consequence: stale authn/authz on remote nodes, bounded ≤10 min, converging on the next successful publication.
### 4. What is your TiDB version? (Required)
```console
Verified live on a 2-node cluster with a fault proxy on the /etcdserverpb.KV/Put service (master).
```
Contributor guide
Research direction
Start by tracing ALTER USER, GRANT, and REVOKE through the local commit and the /tidb/privilege publication using the etcdserverpb.KV/Put path. Inspect how publication errors and retryCnt=0..4 are handled; done means the client sees a failure or publication retries successfully, and remote nodes converge instead of retaining stale privileges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc, sql
- Domain
- authentication, authorization, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 43/100