Existing sessions fail with ERROR 8249 after dropped resource group cleanup
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
This is a follow-up to [#54453](https://github.com/pingcap/tidb/issues/54453), not a regression of the original in-flight request fix.
The original fix is intended to avoid interrupting requests that are already executing when a resource group is dropped. This report covers a different lifecycle case: an existing SQL session remains connected after its resource group is dropped, and then issues a new statement after the PD Resource Controller has cleaned up the tombstone controller.
The original reproduction and the detailed v8.5.7 reproduction are in [this comment](https://github.com/pingcap/tidb/issues/54453#issuecomment-5395087381).
### 1. Minimal reproduce step
Start an unmodified TiUP Playground:
```bash
tiup playground v8.5.7 --db 1 --pd 1 --kv 1 --tiflash 0 --without-monitor
```
The server reports TiDB v8.5.7 and `tidb_enable_resource_control = ON`.
Create a resource group and a user:
```sql
CREATE DATABASE gtoc8677;
CREATE TABLE gtoc8677.person1 (id INT PRIMARY KEY);
INSERT INTO gtoc8677.person1 VALUES (1), (2);
CREATE RESOURCE GROUP rg1 RU_PER_SEC = 1000;
CREATE USER 'usr1'@'%' IDENTIFIED BY 'testpass' RESOURCE GROUP rg1;
GRANT SELECT ON gtoc8677.* TO 'usr1'@'%';
```
1. In Session A, log in as `usr1` and keep the physical connection open.
2. Verify that a table query succeeds:
```sql
SELECT COUNT(1) FROM gtoc8677.person1;
```
3. In Session B, as `root`, move new logins to `default` and drop the old group:
```sql
ALTER USER 'usr1'@'%' RESOURCE GROUP default;
DROP RESOURCE GROUP rg1;
```
4. Run the table query in Session A immediately. It succeeds while the tombstone controller still exists.
5. Leave Session A idle until the resource-group cleanup runs (about five minutes in the reported reproduction; the `rg1` resource-group metric disappears).
6. Run the same table query again in the still-open Session A.
### 2. What did you expect to see?
The existing SQL session should not become unusable merely because its previously assigned resource group was dropped and its client-side tombstone controller was cleaned up.
The intended behavior should be made explicit, but the session should either:
- transparently fall back/rebind to `default`; or
- be invalidated through a deterministic, documented, and actionable session/reconnect path.
### 3. What did you see instead?
The query fails with:
```
ERROR 8249 (HY000): Unknown resource group 'rg1'
```
The physical connection remains alive. `SELECT VERSION()` succeeds on the same connection, while statements that send requests to TiKV fail. The behavior is reproducible with both autocommit and an explicit transaction.
At the time of failure:
- `mysql.user` shows the user mapped to `default`;
- `INFORMATION_SCHEMA.PROCESSLIST.RESOURCE_GROUP` for the existing session remains `rg1`;
- the session's old resource-group name is sent to the client-side resource-control interceptor.
### 4. Root-cause hypothesis
In the PD client version used by v8.5.7 (`github.com/tikv/pd/client` at `b01426f6b08b`):
1. Dropping `rg1` replaces its cached controller with a default-backed tombstone controller.
2. `OnRequestWait` uses that controller while the tombstone is cached.
3. Cleanup deletes the tombstone controller.
4. The existing session still requests `rg1`; the cache misses, PD reports that the group no longer exists, and the request path has no fallback to `default`.
5. client-go returns the error before sending the TiKV RPC, which TiDB maps to ERROR 8249.
The existing PD client test covers fallback while the tombstone entry is still cached, but does not cover tombstone cleanup followed by a request from an existing session. The `fastCleanup` failpoint can make this regression test deterministic.
This should be treated as a resource-group lifecycle/session semantics issue, separate from the original #54453 in-flight request guarantee.
Contributor guide
Research direction
Start with the existing PD client resource-group fallback test and the fastCleanup failpoint described in the issue, then trace how an existing session’s resource-group name reaches client-go after tombstone cleanup. Add deterministic coverage for a post-cleanup request and verify the selected fallback or documented invalidation and reconnect behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100