matrixorigin / matrixorigin/matrixone
[Network Partition] Proxy connManager retains stale CN UUID keys after HAKeeper eviction
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
Proxy's `connManager.cnTunnels` map retains entries for CN nodes that have been expired by HAKeeper and removed from ClusterService routing. These stale keys accumulate with each CN churn event and are never garbage-collected.
## Root Cause
`proxy/conn_manager.go`:
```go
// selectOne() uses cnTunnels map for minCount calculation
m.cnTunnels[uuid] // ← key persists after CN is removed from ClusterService
// disconnect() cleans the tunnelSet contents but not the map key
func (m *connManager) disconnect(...) {
...
tunnels.del(tunnel)
}
```
Contrast with `scaling.doScaling()` which only processes `WorkState == Draining`:
```go
case WorkState_Draining:
// migrate tunnels
// case WorkState_Working:
// case expired: ← never handled
```
## Impact
- Stale UUID keys with empty tunnelSets remain in `m.cnTunnels` indefinitely
- Each residual entry is ~tens of bytes (empty map + UUID string key)
- `selectOne()` minCount calculation may reference stale CN UUIDs (though routing itself is unaffected since ClusterService won't return expired CNs)
- Only materializes under frequent CN churn (rolling restarts, scale-in)
## Trigger Condition
- CN node expires (30s heartbeat gap) → HAKeeper removes it
- Clients disconnecting over time → tunnels drained to zero
- But the map key for the expired CN UUID persists
## Suggested Fix
1. In `ClusterService.refresh()` remove handler, also call `connManager.removeCN(uuid)` to delete the map key
2. Or: periodic sweep of `cnTunnels` keys against current ClusterService CN list
## Severity: LOW
Impact limited to small memory accumulation. No routing correctness issue.
Contributor guide
Assessment
This issue has not been assessed yet.