matrixorigin / matrixorigin/matrixone

[Bug]: HAKeeper panics when replica removal has no target peer

Open
#28,068 4 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Problem

HAKeeper can panic while building a remove-replica operator when the desired target peer set is empty.

Observed twice on `10.222.1.55` while restarting a local launch deployment with persisted test state. The process exits before CN recovery completes.

```text
panic: runtime error: index out of range [0] with length 0

github.com/matrixorigin/matrixone/pkg/hakeeper/operator.(*Builder).buildToRemoveSteps
pkg/hakeeper/operator/builder.go:239
github.com/matrixorigin/matrixone/pkg/hakeeper/operator.(*Builder).buildSteps
pkg/hakeeper/operator/builder.go:312
github.com/matrixorigin/matrixone/pkg/hakeeper/operator.(*Builder).Build
pkg/hakeeper/operator/builder.go:185
github.com/matrixorigin/matrixone/pkg/hakeeper/operator.CreateRemoveReplica
pkg/hakeeper/operator/create_operator.go:36
github.com/matrixorigin/matrixone/pkg/hakeeper/checkers/logservice.(*checkToRemove).check
pkg/hakeeper/checkers/logservice/check.go:321
```

## Root cause

`buildToRemoveSteps` builds and sorts `targets` from `b.targetPeers`, then unconditionally uses `targets[0]` while `b.toRemove` is non-empty. A transition from a non-empty origin set to an empty target set therefore panics instead of returning a reconciliation error or selecting a valid command target.

`buildToRemoveNonVotingSteps` has the same shape and the same unchecked `targets[0]` access, so the fix and tests should cover both voting and non-voting removals.

## Required behavior

- A legal removal of the final replica must have an explicitly defined command target and produce a valid operator, or be rejected with an error before indexing.
- Invalid or transient empty target topology must not crash HAKeeper.
- Voting and non-voting paths must follow the same invariant.
- Add deterministic builder tests for non-empty origin to empty target, mixed add/remove, and ordinary non-empty target controls.

## Reproduction evidence

The persisted test state reproduced the same panic immediately after restart. Moving that exact local test data aside eliminated this panic; a separate clean-cluster bootstrap timeout then occurred, so the panic is tied to the persisted topology transition rather than the SQL statement running at the time of the first observation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.