apache / apache/fluss

[server] Table deletion stuck permanently when StopReplica request fails

Open
#3,357 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

## Summary

When a table is deleted and the `StopReplicaRequest` fails to reach the target TabletServer (e.g., due to network issues or TabletServer being down), the deletion process gets permanently stuck. The replica state remains in `ReplicaDeletionStarted`, blocking any retry. This causes `fluss_coordinator_tableCount` to be higher than the actual tables in ZooKeeper, with no self-healing mechanism.

## Root Cause

### The Deletion State Machine

```
OnlineReplica → OfflineReplica → ReplicaDeletionStarted → ReplicaDeletionSuccessful → NonExistentReplica
```

The `completeDeleteTable` (which calls `coordinatorContext.removeTable()` and decreases tableCount) is only invoked when **all** replicas reach `ReplicaDeletionSuccessful`.

### The Deadlock

In `CoordinatorRequestBatch.sendStopRequest()` (line 465-469):

```java
if (throwable != null) {
// todo: in FLUSS-55886145, we will introduce a sender thread to send the request.
// in here, we just ignore the error.
LOG.warn("Failed to send stop replica request to tablet server {}.", serverId, throwable);
return; // ← No event produced, no state transition
}
```

When the request **fails to send** (network error, TabletServer down):
1. No `DeleteReplicaResponseReceivedEvent` is produced
2. Replica state stays at `ReplicaDeletionStarted` permanently
3. `isEligibleForDeletion()` checks `!coordinatorContext.isAnyReplicaInState(tableId, ReplicaDeletionStarted)` — returns **false**
4. Even if `resumeDeletions()` is called later, it cannot retry because the eligibility check fails
5. **Permanent deadlock** — no code path can move the replica out of `ReplicaDeletionStarted`

### Contrast with Successful Response Error

When the request is sent successfully but the TabletServer returns an error response, the flow works correctly:
- `DeleteReplicaResponseReceivedEvent` is produced
- `retryDeleteAndSuccessDeleteReplicas` handles the failure
- Replica state transitions to allow retry

The bug is specifically in the **request send failure** path (network-level failure or tablet server down).

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CoordinatorRequestBatch.sendStopRequest(), especially the throwable path described around lines 465-469, then trace the replica deletion state machine and resumeDeletions(). Verify how DeleteReplicaResponseReceivedEvent and retryDeleteAndSuccessDeleteReplicas handle failures. Done means a failed StopReplicaRequest can be retried, replicas leave ReplicaDeletionStarted, and table deletion completes with tableCount corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.