pingcap / pingcap/tidb

Admin check failed when inject network partition between tikv pods

Open
#59,562 2 comments 0 reactions 0 assignees View on GitHub
severity/minor sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Deploy 5 tikv nodes, 3 pd nodes and 1 tidb node in K8S.

1. running transaction workload.
2. some online DDL threads randomly add index or drop index.
3. network partition: [basic-tikv-0, basic-tikv-3, basic-tikv-4], [basic-tikv-1, basic-tikv-2]
4. network partition recovery before the transaction workload finish.
5. admin check table but failed.

The admin check failed statements happen after network partition recovery.

more details can get from the following files.

[decimal_too_short.zip](https://github.com/user-attachments/files/18813402/decimal_too_short.zip)

in zip:

> - schema_and_init_data: initial schema, state and data.
> - transaction_workload: execute concurrently.
> - ddl_sequence: DDL threads execute during transaction workload.
> - fault_injection: inject network partition.
> - tidb_logs: tidb and tikv logs.

table2 schema:

```sql
CREATE TABLE `table2` (
`pkId` int(11) DEFAULT NULL,
`pkAttr0` int(11) NOT NULL,
`commonAttr0_0` double(12,3) DEFAULT NULL,
`commonAttr1_0` int(11) DEFAULT NULL,
`commonAttr2_0` decimal(10,2) DEFAULT NULL,
`commonAttr3_0` varchar(10) DEFAULT NULL,
`commonAttr4_0` decimal(10,2) DEFAULT NULL,
PRIMARY KEY (`pkAttr0`) /*T![clustered_index] NONCLUSTERED */,
KEY `table2index_pk` (`pkAttr0`),
KEY `table2index_commAttr2` (`commonAttr2_0`),
KEY `table2index_commAttr4` (`commonAttr4_0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
```

### 2. What did you expect to see? (Required)

After network partition recovery:
```sql
admin check table table2;
-- ok

select pkId from table2;
-- ok

select commonAttr2_0 from table2;
-- ok

select pkId, commonAttr2_0 from table2;
-- ok

select * from table2;
-- ok
```

### 3. What did you see instead (Required)

After network partition recovery:
```sql
admin check table table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3

select pkId from table2;
-- ok

select commonAttr2_0 from table2;
-- ok

select pkId, commonAttr2_0 from table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3

select * from table2;
-- ERROR 1105 (HY000): [components/tidb_query_datatype/src/codec/mysql/decimal.rs:2153]: decimal too short: 1 < 3
```

It looks like an assertion failed in TiKV component.

### 4. What is your TiDB version? (Required)

Tidb v5.1.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.