pingcap / pingcap/tidb

Data divergence after identical workload when `merge-schedule-limit` is reduced from 8 to 1

Open
#69,411 2 comments 0 reactions 0 assignees View on GitHub
component/pd contribution may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

# Data divergence after identical workload when `merge-schedule-limit` is reduced from 8 to 1

## Summary

I hit a value-level data inconsistency while comparing two TiDB deployments with the same topology and the same SQL sequence. The two deployments started from the same schema/data and received the same workload and the same operation at the same point. The only intended difference was one configuration value.

The concerning part is that the final schema/metadata digest matched and the final row count matched, but the committed row-value digest was different. This looks like a correctness issue rather than a simple availability issue or a formatting-only hash difference.

## Environment

- TiDB/TiKV/PD version observed from PD store status: `8.5.6`
- Deployment shape: two independent local clusters
- Topology per cluster: 1 PD, 2 TiDB servers, and 3 TiKV stores per cluster
- Comparison style: same schema, same initial rows, same SQL order, same cluster operation point, different configuration value only

## Configuration difference

```text
Component: pd
Parameter: schedule.merge-schedule-limit
Baseline: 8
Changed: 1
```

The effective configuration shape was:

```toml
# baseline cluster
[schedule]
merge-schedule-limit = 8

# comparison cluster
[schedule]
merge-schedule-limit = 1
```

## Reproduction flow

1. Start two independent TiDB clusters with the topology above.
2. Keep the first cluster on the baseline configuration.
3. Start the second cluster with the changed configuration shown above.
4. Create the same database and tables on both clusters.
5. Insert the same deterministic initial data on both clusters.
6. Run the pre-action SQL phase on both clusters.
7. Apply the operation below at the same point in the sequence.
8. Run the mid-action observation SQL phase.
9. Run the post-action transactional SQL phase.
10. Run `ANALYZE TABLE` and aggregate validation queries on both clusters.
11. Compare metadata digest, row count, value sum, and CRC32-based row digest.

Operation used during the run:

```text
PD health/store membership was queried at the same point in both clusters, using PD HTTP health/store endpoints. The operation is read-only and should not affect committed SQL rows.
```

Workload summary:

```text
Schema setup: 4 statements. Workload phases: pre_action: 52 statements across 4 SQL clients (INSERT=44, USE=8); mid_action: 34 statements across 4 SQL clients (EXPLAIN=8, SELECT=18, USE=8); post_action: 120 statements across 4 SQL clients (ANALYZE=8, COMMIT=24, INSERT=24, SELECT=8, START=24, UPDATE=24, USE=8).
```

The workload is deterministic. The post-action phase repeatedly executes `START TRANSACTION`, `UPDATE`, `INSERT ... ON DUPLICATE KEY UPDATE`, and `COMMIT`, followed by `ANALYZE TABLE` and aggregate checks.

## Expected behavior

Both clusters should finish with the same committed logical data state. The configuration difference may affect timing, scheduling, keepalive behavior, store heartbeat behavior, or recovery latency, but it should not cause different committed results for the same deterministic SQL sequence.

## Actual behavior

The two clusters finished the workload without a schema mismatch, but the logical data state was different.

Health check around the mismatch:

```text
Before validation:
all healthy: True
baseline cluster healthy: True
changed-config cluster healthy: True
baseline SQL reachable: True
changed-config SQL reachable: True

After validation:
all healthy: True
baseline cluster healthy: True
changed-config cluster healthy: True
baseline SQL reachable: True
changed-config SQL reachable: True
```

Observed metadata and data digests:

```text
metadata hash, baseline cluster: 51b654db2fb77790f9028a4a4040e99c89052eaa3a1a5818d8968ba5fff0a683
metadata hash, changed-config cluster: 51b654db2fb77790f9028a4a4040e99c89052eaa3a1a5818d8968ba5fff0a683
metadata hashes match: yes

row count, baseline cluster: 68
row count, changed-config cluster: 68
row counts match: yes

data hash, baseline cluster: 77a300079bba6e18a458127cb12ff2791f864d5a9d39f8f5bb388ce09358fea2
data hash, changed-config cluster: 317746b2146614a02571be9cef76240bb8926bf9050c7fd42f59d5588af72b49
data hashes match: no
```

Aggregate output from the logical data check:

```text
baseline cluster: 68 3461391 143517386732
changed-config cluster: 68 3461360 141664641171
```

The three columns are `row_count`, `SUM(v)`, and a CRC32-based row digest. The row count is identical, but both `SUM(v)` and the row digest differ.

## Why this looks serious

This is not just a different execution plan or a different display order:

- both clusters used the same table definitions;
- both clusters used the same initial rows;
- the same SQL statements were sent to both clusters;
- the same operation was inserted at the same point;
- the metadata digest matched;
- the row count matched;
- the value-level digest differed.

The most suspicious area is the interaction between the changed configuration and the operation timing around PD/TiKV observation, recovery, and SQL transaction execution.

## Schema used

```sql
CREATE DATABASE IF NOT EXISTS repro_db;
USE repro_db;
CREATE TABLE IF NOT EXISTS repro_history_0 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
CREATE TABLE IF NOT EXISTS repro_history_1 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
```

## Validation SQL

```sql
-- row/value digest used after the workload
SELECT
COUNT(*) AS row_count,
COALESCE(SUM(v), 0) AS sum_v,
COALESCE(SUM(CRC32(CONCAT_WS('#', run_id, table_idx, k, store_hint, payload, v, history_round))), 0) AS row_digest
FROM (
SELECT 'repro_history_0' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_0 WHERE run_id='run_1'
UNION ALL
SELECT 'repro_history_1' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_1 WHERE run_id='run_1'
) AS all_rows;

-- metadata check used to rule out a schema-only difference
SELECT
TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, COLUMN_TYPE, IS_NULLABLE, COLUMN_KEY, COLUMN_DEFAULT, EXTRA
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME LIKE 'repro_history_%'
ORDER BY TABLE_NAME, ORDINAL_POSITION;
```

## Complete SQL sequence used after schema creation

The SQL below has only the database name, table names, and run identifier normalized; the statement structure, transaction shape, numeric values, JSON usage, timestamp values, generated-column reads, `ANALYZE TABLE`, and aggregate checks are preserved.

```sql
-- phase: pre_action
USE repro_db;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,0,0,'run_1:0:pre_action:0',0,JSON_OBJECT('run','run_1','table_idx',0,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,1,1,'run_1:0:pre_action:1',17,JSON_OBJECT('run','run_1','table_idx',0,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,2,2,'run_1:0:pre_action:2',34,JSON_OBJECT('run','run_1','table_idx',0,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,3,3,'run_1:0:pre_action:3',51,JSON_OBJECT('run','run_1','table_idx',0,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,4,0,'run_1:0:pre_action:4',68,JSON_OBJECT('run','run_1','table_idx',0,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,5,1,'run_1:0:pre_action:5',85,JSON_OBJECT('run','run_1','table_idx',0,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,6,2,'run_1:0:pre_action:6',102,JSON_OBJECT('run','run_1','table_idx',0,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,7,3,'run_1:0:pre_action:7',119,JSON_OBJECT('run','run_1','table_idx',0,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,8,0,'run_1:0:pre_action:8',136,JSON_OBJECT('run','run_1','table_idx',0,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,9,1,'run_1:0:pre_action:9',153,JSON_OBJECT('run','run_1','table_idx',0,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,10,2,'run_1:0:pre_action:10',170,JSON_OBJECT('run','run_1','table_idx',0,'k',10,'mode','data_history'),13.076923,'2021-11-11 10:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,11,3,'run_1:0:pre_action:11',187,JSON_OBJECT('run','run_1','table_idx',0,'k',11,'mode','data_history'),14.384615,'2021-12-12 11:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,12,0,'run_1:0:pre_action:12',204,JSON_OBJECT('run','run_1','table_idx',0,'k',12,'mode','data_history'),15.692308,'2021-01-13 12:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,13,1,'run_1:0:pre_action:13',221,JSON_OBJECT('run','run_1','table_idx',0,'k',13,'mode','data_history'),17.000000,'2021-02-14 13:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,14,2,'run_1:0:pre_action:14',238,JSON_OBJECT('run','run_1','table_idx',0,'k',14,'mode','data_history'),18.307692,'2021-03-15 14:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,15,3,'run_1:0:pre_action:15',255,JSON_OBJECT('run','run_1','table_idx',0,'k',15,'mode','data_history'),19.615385,'2021-04-16 15:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,16,0,'run_1:0:pre_action:16',272,JSON_OBJECT('run','run_1','table_idx',0,'k',16,'mode','data_history'),20.923077,'2021-05-17 16:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,17,1,'run_1:0:pre_action:17',289,JSON_OBJECT('run','run_1','table_idx',0,'k',17,'mode','data_history'),22.230769,'2021-06-18 17:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,18,2,'run_1:0:pre_action:18',306,JSON_OBJECT('run','run_1','table_idx',0,'k',18,'mode','data_history'),23.538462,'2021-07-19 18:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,19,3,'run_1:0:pre_action:19',323,JSON_OBJECT('run','run_1','table_idx',0,'k',19,'mode','data_history'),24.846154,'2021-08-20 19:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,20,0,'run_1:0:pre_action:20',340,JSON_OBJECT('run','run_1','table_idx',0,'k',20,'mode','data_history'),26.153846,'2021-09-21 20:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,21,1,'run_1:0:pre_action:21',357,JSON_OBJECT('run','run_1','table_idx',0,'k',21,'mode','data_history'),27.461538,'2021-10-22 21:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
USE repro_db;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,0,1,'run_1:1:pre_action:0',100000,JSON_OBJECT('run','run_1','table_idx',1,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,1,2,'run_1:1:pre_action:1',100017,JSON_OBJECT('run','run_1','table_idx',1,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,2,3,'run_1:1:pre_action:2',100034,JSON_OBJECT('run','run_1','table_idx',1,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,3,0,'run_1:1:pre_action:3',100051,JSON_OBJECT('run','run_1','table_idx',1,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,4,1,'run_1:1:pre_action:4',100068,JSON_OBJECT('run','run_1','table_idx',1,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,5,2,'run_1:1:pre_action:5',100085,JSON_OBJECT('run','run_1','table_idx',1,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,6,3,'run_1:1:pre_action:6',100102,JSON_OBJECT('run','run_1','table_idx',1,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,7,0,'run_1:1:pre_action:7',100119,JSON_OBJECT('run','run_1','table_idx',1,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,8,1,'run_1:1:pre_action:8',100136,JSON_OBJECT('run','run_1','table_idx',1,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,9,2,'run_1:1:pre_action:9',100153,JSON_OBJECT('run','run_1','table_idx',1,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,10,3,'run_1:1:pre_action:10',100170,JSON_OBJECT('run','run_1','table_idx',1,'k',10,'mode','data_history'),13.076923,'2021-11-11 10:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,11,0,'run_1:1:pre_action:11',100187,JSON_OBJECT('run','run_1','table_idx',1,'k',11,'mode','data_history'),14.384615,'2021-12-12 11:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,12,1,'run_1:1:pre_action:12',100204,JSON_OBJECT('run','run_1','table_idx',1,'k',12,'mode','data_history'),15.692308,'2021-01-13 12:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,13,2,'run_1:1:pre_action:13',100221,JSON_OBJECT('run','run_1','table_idx',1,'k',13,'mode','data_history'),17.000000,'2021-02-14 13:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,14,3,'run_1:1:pre_action:14',100238,JSON_OBJECT('run','run_1','table_idx',1,'k',14,'mode','data_history'),18.307692,'2021-03-15 14:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,15,0,'run_1:1:pre_action:15',100255,JSON_OBJECT('run','run_1','table_idx',1,'k',15,'mode','data_history'),19.615385,'2021-04-16 15:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,16,1,'run_1:1:pre_action:16',100272,JSON_OBJECT('run','run_1','table_idx',1,'k',16,'mode','data_history'),20.923077,'2021-05-17 16:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,17,2,'run_1:1:pre_action:17',100289,JSON_OBJECT('run','run_1','table_idx',1,'k',17,'mode','data_history'),22.230769,'2021-06-18 17:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,18,3,'run_1:1:pre_action:18',100306,JSON_OBJECT('run','run_1','table_idx',1,'k',18,'mode','data_history'),23.538462,'2021-07-19 18:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,19,0,'run_1:1:pre_action:19',100323,JSON_OBJECT('run','run_1','table_idx',1,'k',19,'mode','data_history'),24.846154,'2021-08-20 19:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,20,1,'run_1:1:pre_action:20',100340,JSON_OBJECT('run','run_1','table_idx',1,'k',20,'mode','data_history'),26.153846,'2021-09-21 20:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,21,2,'run_1:1:pre_action:21',100357,JSON_OBJECT('run','run_1','table_idx',1,'k',21,'mode','data_history'),27.461538,'2021-10-22 21:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);

-- phase: mid_action
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_0 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_0 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_0 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_0' ORDER BY ORDINAL_POSITION;
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_1 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_1 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_1 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_1' ORDER BY ORDINAL_POSITION;

-- phase: post_action
USE repro_db;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,122,2,'run_1:0:history:122',2074,JSON_OBJECT('run','run_1','table_idx',0,'k',122,'mode','data_history'),159.538462,'2021-03-11 02:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,123,3,'run_1:0:history:123',2091,JSON_OBJECT('run','run_1','table_idx',0,'k',123,'mode','data_history'),160.846154,'2021-04-12 03:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,124,0,'run_1:0:history:124',2108,JSON_OBJECT('run','run_1','table_idx',0,'k',124,'mode','data_history'),162.153846,'2021-05-13 04:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_0;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_0 WHERE run_id='run_1';
USE repro_db;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,122,3,'run_1:1:history:122',102074,JSON_OBJECT('run','run_1','table_idx',1,'k',122,'mode','data_history'),159.538462,'2021-03-11 02:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,123,0,'run_1:1:history:123',102091,JSON_OBJECT('run','run_1','table_idx',1,'k',123,'mode','data_history'),160.846154,'2021-04-12 03:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,124,1,'run_1:1:history:124',102108,JSON_OBJECT('run','run_1','table_idx',1,'k',124,'mode','data_history'),162.153846,'2021-05-13 04:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_1;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_1 WHERE run_id='run_1';
```

## Notes for debugging

I would first check whether the changed configuration can alter the timing of TiDB-to-TiKV communication, PD store/region observation, or store rejoin in a way that lets one cluster miss or double-apply one of the committed updates. It may also be useful to compare the transaction commit/lock/region logs around the post-action phase, especially around the `UPDATE ... WHERE MOD(k,4)=...` and `INSERT ... ON DUPLICATE KEY UPDATE` statements.

Contributor guide

Open the contributing guide

Research direction

Reproduce the workload on two independent TiDB 8.5.6 clusters with identical topology, changing only PD's schedule.merge-schedule-limit from 8 to 1. Compare the supplied metadata, row-count, SUM(v), and CRC32 row-digest queries after the transaction and ANALYZE phases; done means identical committed row values and validation results.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.