pingcap / pingcap/tidb

Privilege DCL reports success while etcd publication fails — remote nodes keep stale authentication/authorization

Open
#70,771 0 comments 0 reactions 0 assignees View on GitHub
component/privilege found-by-ai impact/func-failure severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. Deploy a 2-node TiDB cluster.
2. Warm the privilege cache on the remote node (e.g. log in with the current credential).
3. Inject a transient etcd-write failure on the `/etcdserverpb.KV/Put` service for the `/tidb/privilege` key (~10s window; a gRPC fault proxy targeting only that service works — a TCP cut on the PD port kills TSO so the DCL cannot commit).
4. Run a DCL (`ALTER USER` / `GRANT` / `REVOKE`) on the other node during the window.

### 2. What did you expect to see? (Required)

Either the DCL fails, or the publication is retried until success; remote nodes converge to the new authentication/authorization.

### 3. What did you see instead (Required)

The DCL returns success to the client while the etcd publication failed (`etcd-cli put kv failed [key=/tidb/privilege]` retryCnt=0..4 then dropped, `notify update privilege failed`). The prewarmed remote node keeps:

- accepting the old password for new logins (stale authentication), and
- executing statements that were just revoked (stale authorization),

until the next successful publication converges (bounded ≤10 min window).

Root cause: privilege DCL commit and cross-node publication are non-atomic — the commit succeeds locally, but the etcd publication failure is not surfaced to the client, so remote nodes operate on a stale privilege handle snapshot.

Production reachability: rare (fault-common × DCL-low-frequency conjunction; the etcd-write fault itself is routine — PD maintenance/upgrade/disk-latency windows happen regularly). Consequence: stale authn/authz on remote nodes, bounded ≤10 min, converging on the next successful publication.

### 4. What is your TiDB version? (Required)

```console
Verified live on a 2-node cluster with a fault proxy on the /etcdserverpb.KV/Put service (master).
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing ALTER USER, GRANT, and REVOKE through the local commit and the /tidb/privilege publication using the etcdserverpb.KV/Put path. Inspect how publication errors and retryCnt=0..4 are handled; done means the client sees a failure or publication retries successfully, and remote nodes converge instead of retaining stale privileges.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc, sql
Domain
authentication, authorization, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.