`FLUSH STATS_DELTA ... CLUSTER` can return success when one visible TiDB target is unreachable
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Tested on a local TiUP cluster using the PR #68159 binary.
1. Start a TiUP cluster with two normal TiDB nodes:
```sh
tiup playground v8.5.1 \
--db 2 \
--pd 1 \
--kv 1 \
--tiflash 0 \
--db.binpath /path/to/tidb-server \
--host 127.0.0.1 \
--db.port 22000 \
--pd.port 20379 \
--kv.port 20160 \
--without-monitor
```
2. Create and analyze a test table from `tidb-0`:
```sql
DROP DATABASE IF EXISTS netpart68159sendfail;
CREATE DATABASE netpart68159sendfail;
CREATE TABLE netpart68159sendfail.t(id INT PRIMARY KEY, v INT, KEY iv(v));
INSERT INTO netpart68159sendfail.t VALUES (1,1),(2,2),(3,3),(4,4),(5,5);
ANALYZE TABLE netpart68159sendfail.t;
```
3. Start an extra TiDB node connected to the same PD.
SQL listens on `127.0.0.1:22002`, but the advertised address is a non-loopback address whose status port is not reachable:
```sh
tidb-server \
-P 22002 \
--store=tikv \
--host=127.0.0.1 \
--status=28082 \
--status-host=127.0.0.1 \
--advertise-address=192.168.2.217 \
--path=127.0.0.1:20379
```
4. Confirm the extra TiDB is visible in `information_schema.cluster_info`:
```text
type instance status_address
tidb 127.0.0.1:22000 127.0.0.1:10080
tidb 127.0.0.1:22001 127.0.0.1:10081
tidb 192.168.2.217:22002 192.168.2.217:28082
```
5. Confirm the advertised status address refuses connections:
```text
dial tcp 192.168.2.217:28082: connect: connection refused
```
6. Insert rows through the extra TiDB node:
```sql
INSERT INTO netpart68159sendfail.t
VALUES (101,101),(102,102),(103,103),(104,104),(105,105);
```
7. Check raw stats metadata from `tidb-0`:
```sql
SELECT version, modify_count, count
FROM mysql.stats_meta
WHERE table_id = (
SELECT tidb_table_id
FROM information_schema.tables
WHERE table_schema='netpart68159sendfail' AND table_name='t'
);
```
Result before cluster flush:
```text
modify_count count
0 5
```
8. Run cluster flush from `tidb-0` with a client timeout longer than TiDB-cop backoff, for example 90 seconds:
```sql
FLUSH STATS_DELTA netpart68159sendfail.t CLUSTER;
```
9. Check warnings and raw stats metadata again:
```sql
SHOW WARNINGS;
SELECT version, modify_count, count
FROM mysql.stats_meta
WHERE table_id = (
SELECT tidb_table_id
FROM information_schema.tables
WHERE table_schema='netpart68159sendfail' AND table_name='t'
);
```
10. Run local flush on the extra TiDB node:
```sql
FLUSH STATS_DELTA netpart68159sendfail.t;
```
11. Check raw stats metadata again from `tidb-0`.
### 2. What did you expect to see? (Required)
If one visible TiDB target cannot be reached during `FLUSH STATS_DELTA ... CLUSTER`, the initiating session should not report clean success.
At minimum, the initiating session should surface that one TiDB target failed, so the user knows the cluster-wide flush was incomplete and can retry after fixing the node/network issue.
### 3. What did you see instead (Required)
The cluster flush returned success after about 41 seconds:
```text
FLUSH STATS_DELTA netpart68159sendfail.t CLUSTER;
-- Query OK
-- duration: 41.216s
```
`SHOW WARNINGS` on the initiating session was empty:
```text
Level Code Message
```
Raw stats metadata did not change after the cluster flush:
```text
modify_count count
0 5
```
Then a local flush on the extra TiDB node changed the stats metadata:
```text
modify_count count
5 10
```
This shows the extra TiDB still had pending local stats delta, and the earlier cluster flush did not flush that node even though it returned success to the initiating session.
The `tidb-0` log also showed both the send failure and the successful broadcast log:
```text
failed to handle send error: [192.168.2.217:28082] ... connect: connection refused
Successfully broadcast query [sql="FLUSH STATS_DELTA `netpart68159sendfail`.`t` CLUSTER"]
```
One note from the same test setup: if the client-side timeout is shorter than the internal TiDB-cop send backoff, for example 30 seconds, the statement can return `context deadline exceeded` before this silent-success path is reached. The successful-but-incomplete behavior was reproduced with a 90 second client timeout.
### 4. What is your TiDB version? (Required)
Tested binary:
```text
Release Version: v8.5.6-19-g627e14dfa7
Git Commit Hash: 627e14dfa73ddaf6ad5de187fe903879c3fe7646
Git Branch: pr-68159-latest
```
SQL check from the tested TiDB nodes:
```text
@@version
8.0.11-TiDB-v8.5.6-19-g627e14dfa7
```
Contributor guide
Assessment
This issue has not been assessed yet.