ClickHouse / ClickHouse/ClickHouse
Keeper: SIGABRT (LOGICAL_ERROR in commit_in_bg) on every reconfig+restart during quorum migration on 25.8 — regression of #72234?
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
> Related to #72234 (closed, fixed by #72584 for 24.10). We reproduced what looks like the same crash on **25.8**, so either that fix regressed or this is a related-but-distinct path. Opening a fresh issue per the closed-issue triage; happy to merge back into #72234 if preferred.
## Describe the bug
While migrating a Keeper voting quorum between two data centres — promoting `learner`s to `participant`s and demoting `participant`s back to `learner`s, one node at a time — **every membership change that involves a node restart** aborts a keeper with `SIGABRT` from the NuRaft background-commit thread (`LOGICAL_ERROR`). Nodes auto-restart and recover, but the crash is consistent, and one node crashed **twice** (the second time during recovery / changelog replay).
## How to reproduce
**Versions (both affected):**
- `25.8.16.34` (official build, git hash `7938087aa80508fb6d40b8bf024d025809a8880c`)
- `25.8.24.21` (official build)
**Cluster:** 6 keepers in one Raft configuration — 3 voters (`participant`) in DC-A + 3 non-voting `learner`s in DC-B. `true`.
**Procedure:** move the voting quorum DC-A → DC-B and back, **one change at a time** (promote-first), each via `clickhouse-keeper-client`:
1. `reconfig remove ""`
2. `reconfig add "server.=:9234;;"`
3. restart that node so it rejoins with the new role.
Across one forward + one rollback walk (12 promote/demote steps) we observed **13 `SIGABRT` events** — roughly one per restart-bearing step. **Steps that change only `priority` via `reconfig add` (no restart) never crash** — only the restart / changelog-replay path does.
## Expected behaviour
Promote/demote + restart cycles complete without the state machine aborting.
## Error message and/or stacktrace
All `Code: 49`, `LOGICAL_ERROR`:
```
KeeperStateMachine: Failed to commit stored log at index : Code: 49.
DB::Exception: Trying to commit a ZXID () which was not preprocessed. (LOGICAL_ERROR)
RaftInstance: background committing thread encounter err
Trying to commit a ZXID () which was not preprocessed, exiting to protect the system
```
Variant:
```
DB::Exception: Trying to commit a ZXID while the next ZXID to commit is . (LOGICAL_ERROR)
```
On the node that crashed twice (during recovery):
```
KeeperStateMachine: Failed to preprocess stored log at index : Code: 49.
DB::Exception: Got new ZXID () smaller or equal to current ZXID (). It's a bug. (LOGICAL_ERROR)
KeeperLogStore: Cannot completely read changelog on path changelog_<...>.bin,
error: Cannot read all data. Bytes read: 25. Bytes expected: 288.
```
Stack trace (`SIGABRT` / signal 6):
```
(version 25.8.16.34 (official build), git hash 7938087aa80508fb6d40b8bf024d025809a8880c)
(from thread ) Received signal 6 (Aborted)
5. ./src/Coordination/KeeperStateManager.cpp:343: ?
6. ./contrib/NuRaft/src/handle_commit.cxx:171: nuraft::raft_server::commit_in_bg()
7. ./src/Coordination/KeeperServer.cpp:322: DB::KeeperServer::KeeperRaftServer::commit_in_bg()
8. ./src/Common/ThreadPool.h:312 / function.h:610
9. ./src/Common/ThreadPool.cpp:812: ThreadPoolImpl::ThreadFromThreadPool::worker()
10. std::__thread_proxy<...>
```
## Additional context
- Each abort was survivable for us — small keeper state (recovery in seconds), and one-at-a-time membership changes kept quorum throughout, so clients were unaffected. On a large-state cluster the crash → changelog/snapshot recovery loop would be much slower, and #55219 suggests it can become a CrashLoop.
- Same signature family: #72234 (this exact promote/demote scenario, "fixed" by #72584 for 24.10), #58948, #42668, #55219.
- **Questions:** is the #72584 fix expected to be present in 25.8.x (⇒ this is a regression)? Is the restart / changelog-replay path covered by that fix, or is it a distinct code path? Happy to provide full per-node journals.
Contributor guide
Assessment
This issue has not been assessed yet.