Data divergence after identical workload when `grpc-keepalive-time` is reduced from 10 to 3

# Data divergence after identical workload when `grpc-keepalive-time` is reduced from 10 to 3

## Summary

I hit a value-level data inconsistency while comparing two TiDB deployments with the same topology and the same SQL sequence. The two deployments started from the same schema/data and received the same workload and the same operation at the same point. The only intended difference was one configuration value.

The concerning part is that the final schema/metadata digest matched and the final row count matched, but the committed row-value digest was different. This looks like a correctness issue rather than a simple availability issue or a formatting-only hash difference.

## Environment

- TiDB/TiKV/PD version observed from PD store status: `8.5.6`
- Deployment shape: two independent local clusters
- Topology per cluster: 1 PD, 2 TiDB servers, and 3 TiKV stores per cluster
- Comparison style: same schema, same initial rows, same SQL order, same cluster operation point, different configuration value only

## Configuration difference

```text
Component: tidb_sql
Parameter: server.grpc-keepalive-time
Baseline: 10
Changed: 3
```

The effective configuration shape was:

```toml
# baseline cluster
[server]
grpc-keepalive-time = 10

# comparison cluster
[server]
grpc-keepalive-time = 3
```

## Reproduction flow

1. Start two independent TiDB clusters with the topology above.
2. Keep the first cluster on the baseline configuration.
3. Start the second cluster with the changed configuration shown above.
4. Create the same database and tables on both clusters.
5. Insert the same deterministic initial data on both clusters.
6. Run the pre-action SQL phase on both clusters.
7. Apply the operation below at the same point in the sequence.
8. Run the mid-action observation SQL phase.
9. Run the post-action transactional SQL phase.
10. Run `ANALYZE TABLE` and aggregate validation queries on both clusters.
11. Compare metadata digest, row count, value sum, and CRC32-based row digest.

Operation used during the run:

```text
A PD region-status request was issued at the same point in both clusters, for example `GET /pd/api/v1/regions` or an equivalent region-status endpoint. This is an observation operation, so it should not change committed SQL data.
```

Workload summary:

```text
Schema setup: 6 statements. Workload phases: pre_action: 56 statements across 4 SQL clients (INSERT=40, USE=16); mid_action: 68 statements across 4 SQL clients (EXPLAIN=16, SELECT=36, USE=16); post_action: 240 statements across 4 SQL clients (ANALYZE=16, COMMIT=48, INSERT=48, SELECT=16, START=48, UPDATE=48, USE=16).
```

The workload is deterministic. The post-action phase repeatedly executes `START TRANSACTION`, `UPDATE`, `INSERT ... ON DUPLICATE KEY UPDATE`, and `COMMIT`, followed by `ANALYZE TABLE` and aggregate checks.

## Expected behavior

Both clusters should finish with the same committed logical data state. The configuration difference may affect timing, scheduling, keepalive behavior, store heartbeat behavior, or recovery latency, but it should not cause different committed results for the same deterministic SQL sequence.

## Actual behavior

The two clusters finished the workload without a schema mismatch, but the logical data state was different.

Health check around the mismatch:

```text
Before validation:
all healthy: True
baseline cluster healthy: True
changed-config cluster healthy: True
baseline SQL reachable: True
changed-config SQL reachable: True

After validation:
all healthy: True
baseline cluster healthy: True
changed-config cluster healthy: True
baseline SQL reachable: True
changed-config SQL reachable: True
```

Observed metadata and data digests:

```text
metadata hash, baseline cluster: a2883cafe95e6bead72946fc8692506a4853c712b48658463205393b1303c30c
metadata hash, changed-config cluster: a2883cafe95e6bead72946fc8692506a4853c712b48658463205393b1303c30c
metadata hashes match: yes

row count, baseline cluster: 88
row count, changed-config cluster: 88
row counts match: yes

data hash, baseline cluster: fdb6b5534f607a56e1446076231f732ca5a0d44e9d77209fe2406548272ffc09
data hash, changed-config cluster: 99486e041bb1915133831ea40e817c10f6dd6db69daa1b17aa3fd269080d66ab
data hashes match: no
```

Aggregate output from the logical data check:

```text
baseline cluster: 88 13299078 174283069368
changed-config cluster: 88 13299059 180008203489
```

The three columns are `row_count`, `SUM(v)`, and a CRC32-based row digest. The row count is identical, but both `SUM(v)` and the row digest differ.

## Why this looks serious

This is not just a different execution plan or a different display order:

- both clusters used the same table definitions;
- both clusters used the same initial rows;
- the same SQL statements were sent to both clusters;
- the same operation was inserted at the same point;
- the metadata digest matched;
- the row count matched;
- the value-level digest differed.

The most suspicious area is the interaction between the changed configuration and the operation timing around PD/TiKV observation, recovery, and SQL transaction execution.

## Schema used

```sql
CREATE DATABASE IF NOT EXISTS repro_db;
USE repro_db;
CREATE TABLE IF NOT EXISTS repro_history_0 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
CREATE TABLE IF NOT EXISTS repro_history_1 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
CREATE TABLE IF NOT EXISTS repro_history_2 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
CREATE TABLE IF NOT EXISTS repro_history_3 (run_id VARCHAR(64) NOT NULL,table_idx INT NOT NULL,k INT NOT NULL,store_hint INT NOT NULL,payload VARCHAR(256) NOT NULL,v BIGINT NOT NULL,j JSON DEFAULT NULL,d DECIMAL(20,6) DEFAULT 0,ts TIMESTAMP NULL DEFAULT NULL,flag TINYINT NOT NULL DEFAULT 0,history_round INT NOT NULL DEFAULT 0,v_mod INT GENERATED ALWAYS AS (v % 97) STORED,PRIMARY KEY(run_id,table_idx,k),KEY idx_store_hint(store_hint),KEY idx_v(v),KEY idx_flag_ts(flag,ts),KEY idx_case_history(run_id,history_round),KEY idx_v_mod(v_mod));
```

## Validation SQL

```sql
-- row/value digest used after the workload
SELECT
COUNT(*) AS row_count,
COALESCE(SUM(v), 0) AS sum_v,
COALESCE(SUM(CRC32(CONCAT_WS('#', run_id, table_idx, k, store_hint, payload, v, history_round))), 0) AS row_digest
FROM (
SELECT 'repro_history_0' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_0 WHERE run_id='run_1'
UNION ALL
SELECT 'repro_history_1' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_1 WHERE run_id='run_1'
UNION ALL
SELECT 'repro_history_2' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_2 WHERE run_id='run_1'
UNION ALL
SELECT 'repro_history_3' AS tbl, run_id, table_idx, k, store_hint, payload, v, history_round FROM repro_history_3 WHERE run_id='run_1'
) AS all_rows;

-- metadata check used to rule out a schema-only difference
SELECT
TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, COLUMN_TYPE, IS_NULLABLE, COLUMN_KEY, COLUMN_DEFAULT, EXTRA
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME LIKE 'repro_history_%'
ORDER BY TABLE_NAME, ORDINAL_POSITION;
```

## Complete SQL sequence used after schema creation

The SQL below has only the database name, table names, and run identifier normalized; the statement structure, transaction shape, numeric values, JSON usage, timestamp values, generated-column reads, `ANALYZE TABLE`, and aggregate checks are preserved.

```sql
-- phase: pre_action
USE repro_db;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,0,0,'run_1:0:pre_action:0',0,JSON_OBJECT('run','run_1','table_idx',0,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,1,1,'run_1:0:pre_action:1',17,JSON_OBJECT('run','run_1','table_idx',0,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,2,2,'run_1:0:pre_action:2',34,JSON_OBJECT('run','run_1','table_idx',0,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,3,3,'run_1:0:pre_action:3',51,JSON_OBJECT('run','run_1','table_idx',0,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,4,0,'run_1:0:pre_action:4',68,JSON_OBJECT('run','run_1','table_idx',0,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,5,1,'run_1:0:pre_action:5',85,JSON_OBJECT('run','run_1','table_idx',0,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,6,2,'run_1:0:pre_action:6',102,JSON_OBJECT('run','run_1','table_idx',0,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,7,3,'run_1:0:pre_action:7',119,JSON_OBJECT('run','run_1','table_idx',0,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,8,0,'run_1:0:pre_action:8',136,JSON_OBJECT('run','run_1','table_idx',0,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,9,1,'run_1:0:pre_action:9',153,JSON_OBJECT('run','run_1','table_idx',0,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
USE repro_db;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,0,1,'run_1:1:pre_action:0',100000,JSON_OBJECT('run','run_1','table_idx',1,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,1,2,'run_1:1:pre_action:1',100017,JSON_OBJECT('run','run_1','table_idx',1,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,2,3,'run_1:1:pre_action:2',100034,JSON_OBJECT('run','run_1','table_idx',1,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,3,0,'run_1:1:pre_action:3',100051,JSON_OBJECT('run','run_1','table_idx',1,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,4,1,'run_1:1:pre_action:4',100068,JSON_OBJECT('run','run_1','table_idx',1,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,5,2,'run_1:1:pre_action:5',100085,JSON_OBJECT('run','run_1','table_idx',1,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,6,3,'run_1:1:pre_action:6',100102,JSON_OBJECT('run','run_1','table_idx',1,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,7,0,'run_1:1:pre_action:7',100119,JSON_OBJECT('run','run_1','table_idx',1,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,8,1,'run_1:1:pre_action:8',100136,JSON_OBJECT('run','run_1','table_idx',1,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,9,2,'run_1:1:pre_action:9',100153,JSON_OBJECT('run','run_1','table_idx',1,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
USE repro_db;
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,0,2,'run_1:2:pre_action:0',200000,JSON_OBJECT('run','run_1','table_idx',2,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,1,3,'run_1:2:pre_action:1',200017,JSON_OBJECT('run','run_1','table_idx',2,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,2,0,'run_1:2:pre_action:2',200034,JSON_OBJECT('run','run_1','table_idx',2,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,3,1,'run_1:2:pre_action:3',200051,JSON_OBJECT('run','run_1','table_idx',2,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,4,2,'run_1:2:pre_action:4',200068,JSON_OBJECT('run','run_1','table_idx',2,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,5,3,'run_1:2:pre_action:5',200085,JSON_OBJECT('run','run_1','table_idx',2,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,6,0,'run_1:2:pre_action:6',200102,JSON_OBJECT('run','run_1','table_idx',2,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,7,1,'run_1:2:pre_action:7',200119,JSON_OBJECT('run','run_1','table_idx',2,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,8,2,'run_1:2:pre_action:8',200136,JSON_OBJECT('run','run_1','table_idx',2,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,9,3,'run_1:2:pre_action:9',200153,JSON_OBJECT('run','run_1','table_idx',2,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
USE repro_db;
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,0,3,'run_1:3:pre_action:0',300000,JSON_OBJECT('run','run_1','table_idx',3,'k',0,'mode','data_history'),0.000000,'2021-01-01 00:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,1,0,'run_1:3:pre_action:1',300017,JSON_OBJECT('run','run_1','table_idx',3,'k',1,'mode','data_history'),1.307692,'2021-02-02 01:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,2,1,'run_1:3:pre_action:2',300034,JSON_OBJECT('run','run_1','table_idx',3,'k',2,'mode','data_history'),2.615385,'2021-03-03 02:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,3,2,'run_1:3:pre_action:3',300051,JSON_OBJECT('run','run_1','table_idx',3,'k',3,'mode','data_history'),3.923077,'2021-04-04 03:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,4,3,'run_1:3:pre_action:4',300068,JSON_OBJECT('run','run_1','table_idx',3,'k',4,'mode','data_history'),5.230769,'2021-05-05 04:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,5,0,'run_1:3:pre_action:5',300085,JSON_OBJECT('run','run_1','table_idx',3,'k',5,'mode','data_history'),6.538462,'2021-06-06 05:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,6,1,'run_1:3:pre_action:6',300102,JSON_OBJECT('run','run_1','table_idx',3,'k',6,'mode','data_history'),7.846154,'2021-07-07 06:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,7,2,'run_1:3:pre_action:7',300119,JSON_OBJECT('run','run_1','table_idx',3,'k',7,'mode','data_history'),9.153846,'2021-08-08 07:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,8,3,'run_1:3:pre_action:8',300136,JSON_OBJECT('run','run_1','table_idx',3,'k',8,'mode','data_history'),10.461538,'2021-09-09 08:00:00',0,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,9,0,'run_1:3:pre_action:9',300153,JSON_OBJECT('run','run_1','table_idx',3,'k',9,'mode','data_history'),11.769231,'2021-10-10 09:00:00',1,0) ON DUPLICATE KEY UPDATE store_hint=VALUES(store_hint),payload=VALUES(payload),v=VALUES(v),j=VALUES(j),d=VALUES(d),ts=VALUES(ts),flag=VALUES(flag),history_round=VALUES(history_round);

-- phase: mid_action
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_0 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_0 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_0 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_0' ORDER BY ORDINAL_POSITION;
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_1 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_1 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_1 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_1' ORDER BY ORDINAL_POSITION;
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_2 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_2 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_2 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_2' ORDER BY ORDINAL_POSITION;
USE repro_db;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(CRC32(CONCAT_WS('#',run_id,table_idx,k,store_hint,payload,v,history_round))),0) FROM repro_history_3 WHERE run_id='run_1';
EXPLAIN SELECT store_hint, COUNT(*), COALESCE(SUM(v),0) FROM repro_history_3 WHERE run_id='run_1' GROUP BY store_hint ORDER BY store_hint;
SELECT COUNT(*), COALESCE(SUM(v_mod),0) FROM repro_history_3 WHERE run_id='run_1';
SELECT TABLE_NAME,COLUMN_NAME,COLUMN_TYPE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='repro_db' AND TABLE_NAME='repro_history_3' ORDER BY ORDINAL_POSITION;

-- phase: post_action
USE repro_db;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,110,2,'run_1:0:history:110',1870,JSON_OBJECT('run','run_1','table_idx',0,'k',110,'mode','data_history'),143.846154,'2021-03-27 14:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,111,3,'run_1:0:history:111',1887,JSON_OBJECT('run','run_1','table_idx',0,'k',111,'mode','data_history'),145.153846,'2021-04-28 15:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_0 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_0(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',0,112,0,'run_1:0:history:112',1904,JSON_OBJECT('run','run_1','table_idx',0,'k',112,'mode','data_history'),146.461538,'2021-05-01 16:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_0;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_0 WHERE run_id='run_1';
USE repro_db;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,110,3,'run_1:1:history:110',101870,JSON_OBJECT('run','run_1','table_idx',1,'k',110,'mode','data_history'),143.846154,'2021-03-27 14:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,111,0,'run_1:1:history:111',101887,JSON_OBJECT('run','run_1','table_idx',1,'k',111,'mode','data_history'),145.153846,'2021-04-28 15:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_1 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_1(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',1,112,1,'run_1:1:history:112',101904,JSON_OBJECT('run','run_1','table_idx',1,'k',112,'mode','data_history'),146.461538,'2021-05-01 16:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_1;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_1 WHERE run_id='run_1';
USE repro_db;
START TRANSACTION;
UPDATE repro_history_2 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,110,0,'run_1:2:history:110',201870,JSON_OBJECT('run','run_1','table_idx',2,'k',110,'mode','data_history'),143.846154,'2021-03-27 14:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_2 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,111,1,'run_1:2:history:111',201887,JSON_OBJECT('run','run_1','table_idx',2,'k',111,'mode','data_history'),145.153846,'2021-04-28 15:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_2 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_2(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',2,112,2,'run_1:2:history:112',201904,JSON_OBJECT('run','run_1','table_idx',2,'k',112,'mode','data_history'),146.461538,'2021-05-01 16:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_2;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_2 WHERE run_id='run_1';
USE repro_db;
START TRANSACTION;
UPDATE repro_history_3 SET v=v+7, history_round=history_round+1, payload=CONCAT(payload, ':h0') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,110,1,'run_1:3:history:110',301870,JSON_OBJECT('run','run_1','table_idx',3,'k',110,'mode','data_history'),143.846154,'2021-03-27 14:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_3 SET v=v+8, history_round=history_round+1, payload=CONCAT(payload, ':h1') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,111,2,'run_1:3:history:111',301887,JSON_OBJECT('run','run_1','table_idx',3,'k',111,'mode','data_history'),145.153846,'2021-04-28 15:00:00',1,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
START TRANSACTION;
UPDATE repro_history_3 SET v=v+9, history_round=history_round+1, payload=CONCAT(payload, ':h2') WHERE run_id='run_1' AND MOD(k,1)=0;
INSERT INTO repro_history_3(run_id,table_idx,k,store_hint,payload,v,j,d,ts,flag,history_round) VALUES('run_1',3,112,3,'run_1:3:history:112',301904,JSON_OBJECT('run','run_1','table_idx',3,'k',112,'mode','data_history'),146.461538,'2021-05-01 16:00:00',0,0) ON DUPLICATE KEY UPDATE v=VALUES(v),payload=VALUES(payload),history_round=history_round+1;
COMMIT;
ANALYZE TABLE repro_history_3;
SELECT COUNT(*), COALESCE(SUM(v),0), COALESCE(SUM(history_round),0) FROM repro_history_3 WHERE run_id='run_1';
```

## Notes for debugging

I would first check whether the changed configuration can alter the timing of TiDB-to-TiKV communication, PD store/region observation, or store rejoin in a way that lets one cluster miss or double-apply one of the committed updates. It may also be useful to compare the transaction commit/lock/region logs around the post-action phase, especially around the `UPDATE ... WHERE MOD(k,4)=...` and `INSERT ... ON DUPLICATE KEY UPDATE` statements.

Start by reproducing the two-cluster workload with TiDB/TiKV/PD 8.5.6 and the two server.grpc-keepalive-time values, including the PD region-status request. Compare the validation SQL results, transaction outcomes, and cluster logs to isolate the source of the value divergence. Done means the cause is identified and identical workloads produce matching committed data.

Written by the indexing model from the issue text.