TiDB SQL node can remain unhealthy after restart when `stores-refresh-interval` is reduced from 60 to 15
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
# TiDB SQL node can remain unhealthy after restart when `stores-refresh-interval` is reduced from 60 to 15
## Summary
I observed a recovery/degradation problem after a management restart with a valid configuration change. The baseline cluster and the changed-config cluster were set up with the same topology and were run through the same basic SQL workload and recovery observation flow. The changed-config cluster was expected to become healthy again after the restart window, but it remained unhealthy while the baseline side was classified as healthy.
This report is about recovery/availability rather than confirmed data corruption. The reason I consider it worth tracking is that the configuration value is a normal TiDB/TiKV configuration value, and restart/rejoin should not leave the cluster persistently degraded under this small workload.
## Environment
- TiDB/TiKV/PD version observed from PD store status: `8.5.6`
- Deployment shape: two independent local clusters
- Topology per cluster: 1 PD, 2 TiDB servers, and 3 TiKV stores per cluster
- Comparison style: same workload and recovery observation, different configuration value only
## Configuration difference
```text
Component: tidb
Parameter: server.stores-refresh-interval
Baseline: 60
Changed: 15
```
Effective configuration shape:
```toml
# baseline cluster
[server]
stores-refresh-interval = 60
# comparison cluster
[server]
stores-refresh-interval = 15
```
## Reproduction flow
1. Start two independent TiDB clusters with the topology above.
2. Keep one cluster on the baseline configuration.
3. Start the other cluster with the changed configuration shown above.
4. Verify both clusters are healthy before the restart step.
5. Run a small SQL workload on both clusters.
6. Trigger the management restart/recovery operation.
7. Wait for the normal recovery/rejoin window.
8. Query SQL health and PD health/store status.
9. Compare the final health classification of the two clusters.
Operation used during the run:
```text
Two TiKV stores were restarted around the middle of the sequence. The restart was used to exercise recovery/rejoin behavior after the configuration change.
```
Workload summary:
```text
The SQL part is intentionally small in this reproduction: 3 statements creating one table, inserting one row, and reading it back.
```
## Expected behavior
Both clusters should recover after the restart/rejoin window. A valid configuration change may alter heartbeat frequency, store refresh latency, or recovery timing, but it should not leave the changed-config cluster persistently unhealthy after the same management restart flow.
Expected health before the operation:
```text
baseline cluster healthy: True
changed-config cluster healthy: True
```
## Actual behavior
Final health classification after the operation:
```text
baseline cluster healthy: True
changed-config cluster healthy: False
```
Additional observation:
```text
In a later runtime snapshot, the SQL endpoint for at least one TiDB process was not reachable and one PD health request failed because the corresponding process was no longer running. That is consistent with a restart/recovery problem rather than a pure SQL error.
```
The comparison immediately before this failure point had the same input properties:
```text
only the configuration value differed: True
same workload: True
same initial data state: True
same timing plan: True
same cluster action template: True
same observation query: True
previous data-state hash, baseline: e6f58ee139f35f17484acd23143e7b0ce35930a615d663be22af8e30c6e108dc
previous data-state hash, changed: e6f58ee139f35f17484acd23143e7b0ce35930a615d663be22af8e30c6e108dc
```
## SQL workload used in the recovery check
```sql
CREATE TABLE IF NOT EXISTS t_replay (id INT PRIMARY KEY, val VARCHAR(100)) ENGINE=TiDB;
INSERT INTO t_replay VALUES (1, 'replay_test');
SELECT * FROM t_replay;
```
## Health checks used
```bash
# SQL reachability
mysql -h -P 4000 -u -p -e "SELECT 1 AS health_check;"
# PD member health
curl -fsS http://:2379/pd/api/v1/health
# PD store view
curl -fsS http://:2379/pd/api/v1/stores
```
## Why this looks relevant
The failure is not caused by changing the SQL workload. The SQL part is intentionally simple, and the comparison point was designed to isolate the configuration/recovery behavior. The most likely area to inspect is the component that consumes the changed setting and its interaction with restart/rejoin logic, PD store state, TiDB store-cache refresh, or TiKV heartbeat propagation.
## Suggested area to inspect
- whether the changed setting can delay or suppress store-state refresh after restart;
- whether PD marks stores as disconnected for longer than expected;
- whether TiDB SQL nodes keep stale store status after restart;
- whether the recovery path has a race between restart completion and store/region health observation.
Contributor guide
Research direction
Start by reproducing the two-cluster comparison with `server.stores-refresh-interval` set to 60 and 15, then inspect SQL reachability and the PD `/pd/api/v1/health` and `/pd/api/v1/stores` endpoints after the TiKV restart. Trace the setting's consumers and restart/rejoin health flow; done means the changed-config cluster recovers and reports healthy like the baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100