[server] Table deletion stuck permanently when StopReplica request fails
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
## Summary
When a table is deleted and the `StopReplicaRequest` fails to reach the target TabletServer (e.g., due to network issues or TabletServer being down), the deletion process gets permanently stuck. The replica state remains in `ReplicaDeletionStarted`, blocking any retry. This causes `fluss_coordinator_tableCount` to be higher than the actual tables in ZooKeeper, with no self-healing mechanism.
## Root Cause
### The Deletion State Machine
```
OnlineReplica → OfflineReplica → ReplicaDeletionStarted → ReplicaDeletionSuccessful → NonExistentReplica
```
The `completeDeleteTable` (which calls `coordinatorContext.removeTable()` and decreases tableCount) is only invoked when **all** replicas reach `ReplicaDeletionSuccessful`.
### The Deadlock
In `CoordinatorRequestBatch.sendStopRequest()` (line 465-469):
```java
if (throwable != null) {
// todo: in FLUSS-55886145, we will introduce a sender thread to send the request.
// in here, we just ignore the error.
LOG.warn("Failed to send stop replica request to tablet server {}.", serverId, throwable);
return; // ← No event produced, no state transition
}
```
When the request **fails to send** (network error, TabletServer down):
1. No `DeleteReplicaResponseReceivedEvent` is produced
2. Replica state stays at `ReplicaDeletionStarted` permanently
3. `isEligibleForDeletion()` checks `!coordinatorContext.isAnyReplicaInState(tableId, ReplicaDeletionStarted)` — returns **false**
4. Even if `resumeDeletions()` is called later, it cannot retry because the eligibility check fails
5. **Permanent deadlock** — no code path can move the replica out of `ReplicaDeletionStarted`
### Contrast with Successful Response Error
When the request is sent successfully but the TabletServer returns an error response, the flow works correctly:
- `DeleteReplicaResponseReceivedEvent` is produced
- `retryDeleteAndSuccessDeleteReplicas` handles the failure
- Replica state transitions to allow retry
The bug is specifically in the **request send failure** path (network-level failure or tablet server down).
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with CoordinatorRequestBatch.sendStopRequest(), especially the throwable path described around lines 465-469, then trace the replica deletion state machine and resumeDeletions(). Verify how DeleteReplicaResponseReceivedEvent and retryDeleteAndSuccessDeleteReplicas handle failures. Done means a failed StopReplicaRequest can be retried, replicas leave ReplicaDeletionStarted, and table deletion completes with tableCount corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100