apache / apache/fluss

Coordinator can advertise a leader before the target tablet server admits the replica, causing NotLeaderOrFollowerException to recur for minutes after a large-table restart

Open
#4,149 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

The coordinator maintains two independently-updated views of replica state:

- `TabletServerMetadataCache` (pushed via `UpdateMetadataRequest`) — determines what leader info clients see via `metadata()`.
- `ReplicaManager.allReplicas` (activated via `NotifyLeaderAndIsrRequest`, `ReplicaManager#maybeCreateReplica`) — determines whether a tablet server can actually serve `fetchLog`/`putKv` for a bucket.

These are sent as two separate RPCs (`CoordinatorRequestBatch#sendRequestToTabletServers`) with no atomicity guarantee between when the coordinator decides a new leader and when the target tablet server has actually admitted the replica.

Under normal conditions this window is milliseconds and effectively unobservable. For large tables, however, a restarting tablet server can take minutes to complete local log recovery (observed: a single 12-bucket table taking ~11 minutes) before it gets to processing `NotifyLeaderAndIsrRequest`, while `UpdateMetadataRequest` can be broadcast to clients well before that (e.g. as part of `CoordinatorEventProcessor#processNewTabletServer` when the server re-registers).

A client that picks up the new leader from metadata during this window sends requests to a server whose `ReplicaManager` still treats the bucket as `NoneReplica`. The server responds with `NotLeaderOrFollowerException` (`ReplicaManager#getReplicaOrException`), the client refreshes metadata (which still shows the same, still-not-actually-ready leader), and the cycle repeats — a tight retry loop reproducing hundreds to thousands of errors per minute — until the tablet server finally catches up. In our testing this recurred for anywhere from several minutes to over ten minutes after the corresponding tablet server pod already reported `Running`/ready at the Kubernetes level.

Symptom in client logs (verbatim):
```
ERROR org.apache.fluss.client.table.scanner.log.LogFetcher - Failed to fetch log from node 1 for bucket TableBucket{tableId=3, bucket=2}
org.apache.fluss.exception.NotLeaderOrFollowerException: Not leader or follower.
WARN org.apache.fluss.client.table.scanner.log.LogFetcher - Invalid metadata error in fetch log request. Going to request metadata update.
```

The coordinator side is completely quiet during this window — no errors, no failed leader elections, ZooKeeper's `leader_isr` data is correct throughout. This is a pure client-observable symptom; the coordinator's authoritative state is never actually corrupted, it's just prematurely advertised.

### Reproduction

1. Create a table with a large amount of data (large enough that a tablet server restart takes multiple minutes to complete local log recovery for its buckets).
2. Restart the tablet server hosting that table's leader/follower replicas (e.g. as part of a rolling update).
3. Have a client (e.g. a continuously-running scan/tiering job) keep reading from the affected buckets throughout the restart.
4. Observe `NotLeaderOrFollowerException`/`LeaderNotAvailableException` continuing to recur on the client well after the tablet server pod reports ready, until the coordinator happens to trigger another `NotifyLeaderAndIsrRequest`/`UpdateMetadataRequest` round for the affected buckets.

Small tables (log recovery completing in seconds) are unlikely to trigger this in practice — the window is too short to reliably observe.

### Solution

I have a fix and will submit a PR shortly. Summary: the coordinator already has a `pendingLeaderActivationBuckets` mechanism in `CoordinatorContext` that tracks buckets awaiting `NotifyLeaderAndIsrRequest` acknowledgement, but it was previously only consulted by the read-only Cluster Health API. The fix wires this into `CoordinatorRequestBatch#addUpdateMetadataRequestForTabletServers` so that a bucket's leader is withheld from `UpdateMetadataRequest` for as long as it is marked pending, and the mark is only cleared once the corresponding `NotifyLeaderAndIsrRequest` is genuinely acknowledged by the target server (or the replica goes offline via re-election). This aligns "client-visible leader" with "server-admitted leader" without needing any client-side retry/backoff changes.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CoordinatorRequestBatch#addUpdateMetadataRequestForTabletServers and CoordinatorContext#pendingLeaderActivationBuckets, then trace NotifyLeaderAndIsrRequest handling through ReplicaManager#maybeCreateReplica and CoordinatorEventProcessor#processNewTabletServer. Done means metadata withholds a leader until the target acknowledges replica activation, while re-election or an offline replica clears the pending state, preventing recurring client errors during recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.