Admin check failed when inject network partition between tikv pods
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Deploy 5 tikv nodes, 3 pd nodes and 1 tidb node in K8S.
1. running transaction workload.
2. some online DDL threads randomly add index or drop index.
3. network partition: [basic-tikv-0, basic-tikv-3, basic-tikv-4], [basic-tikv-1, basic-tikv-2]
4. network partition recovery before the transaction workload finish.
5. admin check table but failed.
The admin check failed statements happen after network partition recovery.
more details can get from the following files.
[decimal_too_short.zip](https://github.com/user-attachments/files/18813402/decimal_too_short.zip)
in zip:
> - schema_and_init_data: initial schema, state and data.
> - transaction_workload: execute concurrently.
> - ddl_sequence: DDL threads execute during transaction workload.
> - fault_injection: inject network partition.
> - tidb_logs: tidb and tikv logs.
table2 schema:
```sql
CREATE TABLE `table2` (
`pkId` int(11) DEFAULT NULL,
`pkAttr0` int(11) NOT NULL,
`commonAttr0_0` double(12,3) DEFAULT NULL,
`commonAttr1_0` int(11) DEFAULT NULL,
`commonAttr2_0` decimal(10,2) DEFAULT NULL,
`commonAttr3_0` varchar(10) DEFAULT NULL,
`commonAttr4_0` decimal(10,2) DEFAULT NULL,
PRIMARY KEY (`pkAttr0`) /*T![clustered_index] NONCLUSTERED */,
KEY `table2index_pk` (`pkAttr0`),
KEY `table2index_commAttr2` (`commonAttr2_0`),
KEY `table2index_commAttr4` (`commonAttr4_0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
```
### 2. What did you expect to see? (Required)
After network partition recovery:
```sql
admin check table table2;
-- ok
select pkId from table2;
-- ok
select commonAttr2_0 from table2;
-- ok
select pkId, commonAttr2_0 from table2;
-- ok
select * from table2;
-- ok
```
### 3. What did you see instead (Required)
After network partition recovery:
```sql
admin check table table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3
select pkId from table2;
-- ok
select commonAttr2_0 from table2;
-- ok
select pkId, commonAttr2_0 from table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3
select * from table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3
```
It looks like an assertion failed in TiKV component.
### 4. What is your TiDB version? (Required)
Tidb v5.1.2
Contributor guide
Assessment
This issue has not been assessed yet.