matrixorigin / matrixorigin/matrixone

[Network Partition] Proxy connManager retains stale CN UUID keys after HAKeeper eviction

Open
#25,208 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

Proxy's `connManager.cnTunnels` map retains entries for CN nodes that have been expired by HAKeeper and removed from ClusterService routing. These stale keys accumulate with each CN churn event and are never garbage-collected.

## Root Cause

`proxy/conn_manager.go`:

```go
// selectOne() uses cnTunnels map for minCount calculation
m.cnTunnels[uuid] // ← key persists after CN is removed from ClusterService

// disconnect() cleans the tunnelSet contents but not the map key
func (m *connManager) disconnect(...) {
...
tunnels.del(tunnel)
}
```

Contrast with `scaling.doScaling()` which only processes `WorkState == Draining`:

```go
case WorkState_Draining:
// migrate tunnels
// case WorkState_Working:
// case expired: ← never handled
```

## Impact

- Stale UUID keys with empty tunnelSets remain in `m.cnTunnels` indefinitely
- Each residual entry is ~tens of bytes (empty map + UUID string key)
- `selectOne()` minCount calculation may reference stale CN UUIDs (though routing itself is unaffected since ClusterService won't return expired CNs)
- Only materializes under frequent CN churn (rolling restarts, scale-in)

## Trigger Condition

- CN node expires (30s heartbeat gap) → HAKeeper removes it
- Clients disconnecting over time → tunnels drained to zero
- But the map key for the expired CN UUID persists

## Suggested Fix

1. In `ClusterService.refresh()` remove handler, also call `connManager.removeCN(uuid)` to delete the map key
2. Or: periodic sweep of `cnTunnels` keys against current ClusterService CN list

## Severity: LOW

Impact limited to small memory accumulation. No routing correctness issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.