TiKV stores can remain disconnected after restart when `pd-store-heartbeat-tick-interval` is increased from 20s to 60s
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
# TiKV stores can remain disconnected after restart when `pd-store-heartbeat-tick-interval` is increased from 20s to 60s
## Summary
I observed a recovery/degradation problem after a management restart with a valid configuration change. The baseline cluster and the changed-config cluster were set up with the same topology and were run through the same basic SQL workload and recovery observation flow. The changed-config cluster was expected to become healthy again after the restart window, but it remained unhealthy while the baseline side was classified as healthy.
This report is about recovery/availability rather than confirmed data corruption. The reason I consider it worth tracking is that the configuration value is a normal TiDB/TiKV configuration value, and restart/rejoin should not leave the cluster persistently degraded under this small workload.
## Environment
- TiDB/TiKV/PD version observed from PD store status: `8.5.6`
- Deployment shape: two independent local clusters
- Topology per cluster: 1 PD, 2 TiDB servers, and 3 TiKV stores per cluster
- Comparison style: same workload and recovery observation, different configuration value only
## Configuration difference
```text
Component: tikv
Parameter: raftstore.pd-store-heartbeat-tick-interval
Baseline: 20s
Changed: 60s
```
Effective configuration shape:
```toml
# baseline cluster
[raftstore]
pd-store-heartbeat-tick-interval = "20s"
# comparison cluster
[raftstore]
pd-store-heartbeat-tick-interval = "60s"
```
## Reproduction flow
1. Start two independent TiDB clusters with the topology above.
2. Keep one cluster on the baseline configuration.
3. Start the other cluster with the changed configuration shown above.
4. Verify both clusters are healthy before the restart step.
5. Run a small SQL workload on both clusters.
6. Trigger the management restart/recovery operation.
7. Wait for the normal recovery/rejoin window.
8. Query SQL health and PD health/store status.
9. Compare the final health classification of the two clusters.
Operation used during the run:
```text
One TiKV store was restarted and the cluster was expected to rejoin and become healthy again after the recovery window.
```
Workload summary:
```text
The SQL part is intentionally small in this reproduction: 3 statements creating one table, inserting one row, and reading it back.
```
## Expected behavior
Both clusters should recover after the restart/rejoin window. A valid configuration change may alter heartbeat frequency, store refresh latency, or recovery timing, but it should not leave the changed-config cluster persistently unhealthy after the same management restart flow.
Expected health before the operation:
```text
baseline cluster healthy: True
changed-config cluster healthy: True
```
## Actual behavior
Final health classification after the operation:
```text
baseline cluster healthy: True
changed-config cluster healthy: False
```
Additional observation:
```text
The PD endpoint itself was reachable, but PD reported multiple TiKV stores as `Disconnected` on the changed-config cluster after the restart window. SQL health checks also failed with `ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3306' (111)` in the captured health check.
```
The comparison immediately before this failure point had the same input properties:
```text
only the configuration value differed: True
same workload: True
same initial data state: True
same timing plan: True
same cluster action template: True
same observation query: True
previous data-state hash, baseline: ce8693ff91342197719667689c8594fdbbd51ff08127afc611ebe933aa1b8f61
previous data-state hash, changed: ce8693ff91342197719667689c8594fdbbd51ff08127afc611ebe933aa1b8f61
```
## SQL workload used in the recovery check
```sql
CREATE TABLE IF NOT EXISTS t_replay (id INT PRIMARY KEY, val VARCHAR(100)) ENGINE=TiDB;
INSERT INTO t_replay VALUES (1, 'replay_test');
SELECT * FROM t_replay;
```
## Health checks used
```bash
# SQL reachability
mysql -h -P 4000 -u -p -e "SELECT 1 AS health_check;"
# PD member health
curl -fsS http://:2379/pd/api/v1/health
# PD store view
curl -fsS http://:2379/pd/api/v1/stores
```
## Why this looks relevant
The failure is not caused by changing the SQL workload. The SQL part is intentionally simple, and the comparison point was designed to isolate the configuration/recovery behavior. The most likely area to inspect is the component that consumes the changed setting and its interaction with restart/rejoin logic, PD store state, TiDB store-cache refresh, or TiKV heartbeat propagation.
## Suggested area to inspect
- whether the changed setting can delay or suppress store-state refresh after restart;
- whether PD marks stores as disconnected for longer than expected;
- whether TiDB SQL nodes keep stale store status after restart;
- whether the recovery path has a race between restart completion and store/region health observation.
Contributor guide
Research direction
Start by reproducing the two-cluster comparison with raftstore.pd-store-heartbeat-tick-interval set to 20s and 60s, then run the listed MySQL and PD health checks after restarting one TiKV store. Inspect PD store status, TiDB store-cache refresh, and TiKV heartbeat/rejoin behavior around the recovery window. Done means both clusters recover consistently, SQL health succeeds, and PD reports all TiKV stores as connected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100