Coordinator can advertise a leader before the target tablet server admits the replica, causing NotLeaderOrFollowerException to recur for minutes after a large-table restart
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
The coordinator maintains two independently-updated views of replica state:
- `TabletServerMetadataCache` (pushed via `UpdateMetadataRequest`) — determines what leader info clients see via `metadata()`.
- `ReplicaManager.allReplicas` (activated via `NotifyLeaderAndIsrRequest`, `ReplicaManager#maybeCreateReplica`) — determines whether a tablet server can actually serve `fetchLog`/`putKv` for a bucket.
These are sent as two separate RPCs (`CoordinatorRequestBatch#sendRequestToTabletServers`) with no atomicity guarantee between when the coordinator decides a new leader and when the target tablet server has actually admitted the replica.
Under normal conditions this window is milliseconds and effectively unobservable. For large tables, however, a restarting tablet server can take minutes to complete local log recovery (observed: a single 12-bucket table taking ~11 minutes) before it gets to processing `NotifyLeaderAndIsrRequest`, while `UpdateMetadataRequest` can be broadcast to clients well before that (e.g. as part of `CoordinatorEventProcessor#processNewTabletServer` when the server re-registers).
A client that picks up the new leader from metadata during this window sends requests to a server whose `ReplicaManager` still treats the bucket as `NoneReplica`. The server responds with `NotLeaderOrFollowerException` (`ReplicaManager#getReplicaOrException`), the client refreshes metadata (which still shows the same, still-not-actually-ready leader), and the cycle repeats — a tight retry loop reproducing hundreds to thousands of errors per minute — until the tablet server finally catches up. In our testing this recurred for anywhere from several minutes to over ten minutes after the corresponding tablet server pod already reported `Running`/ready at the Kubernetes level.
Symptom in client logs (verbatim):
```
ERROR org.apache.fluss.client.table.scanner.log.LogFetcher - Failed to fetch log from node 1 for bucket TableBucket{tableId=3, bucket=2}
org.apache.fluss.exception.NotLeaderOrFollowerException: Not leader or follower.
WARN org.apache.fluss.client.table.scanner.log.LogFetcher - Invalid metadata error in fetch log request. Going to request metadata update.
```
The coordinator side is completely quiet during this window — no errors, no failed leader elections, ZooKeeper's `leader_isr` data is correct throughout. This is a pure client-observable symptom; the coordinator's authoritative state is never actually corrupted, it's just prematurely advertised.
### Reproduction
1. Create a table with a large amount of data (large enough that a tablet server restart takes multiple minutes to complete local log recovery for its buckets).
2. Restart the tablet server hosting that table's leader/follower replicas (e.g. as part of a rolling update).
3. Have a client (e.g. a continuously-running scan/tiering job) keep reading from the affected buckets throughout the restart.
4. Observe `NotLeaderOrFollowerException`/`LeaderNotAvailableException` continuing to recur on the client well after the tablet server pod reports ready, until the coordinator happens to trigger another `NotifyLeaderAndIsrRequest`/`UpdateMetadataRequest` round for the affected buckets.
Small tables (log recovery completing in seconds) are unlikely to trigger this in practice — the window is too short to reliably observe.
### Solution
I have a fix and will submit a PR shortly. Summary: the coordinator already has a `pendingLeaderActivationBuckets` mechanism in `CoordinatorContext` that tracks buckets awaiting `NotifyLeaderAndIsrRequest` acknowledgement, but it was previously only consulted by the read-only Cluster Health API. The fix wires this into `CoordinatorRequestBatch#addUpdateMetadataRequestForTabletServers` so that a bucket's leader is withheld from `UpdateMetadataRequest` for as long as it is marked pending, and the mark is only cleared once the corresponding `NotifyLeaderAndIsrRequest` is genuinely acknowledged by the target server (or the replica goes offline via re-election). This aligns "client-visible leader" with "server-admitted leader" without needing any client-side retry/backoff changes.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with CoordinatorRequestBatch#addUpdateMetadataRequestForTabletServers and CoordinatorContext#pendingLeaderActivationBuckets, then trace NotifyLeaderAndIsrRequest handling through ReplicaManager#maybeCreateReplica and CoordinatorEventProcessor#processNewTabletServer. Done means metadata withholds a leader until the target acknowledges replica activation, while re-election or an offline replica clears the pending state, preventing recurring client errors during recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100