[client] NotLeaderOrFollowerException persists when a stale TabletServer endpoint is reused
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.8.0
### Please describe the bug 🐞
During a rolling upgrade, a TabletServer may restart with a new endpoint, while its old endpoint is reused by another TabletServer. A client may then keep a cached connection for the original server ID that actually connects to the other TabletServer.
A request such as `listOffsets` reaches the wrong server and receives `NotLeaderOrFollowerException`. The current handling completes the future exceptionally without disconnecting the cached connection or forcing a cluster metadata refresh. Subsequent caller-side retries reuse the same connection, making the failure persistent.
This is related to #2097 and #3389. Unlike the transient leader-change case described in #3389, the stale endpoint remains reachable because it now belongs to another TabletServer, so no `NetworkException` is raised to invalidate the connection.
### Steps to reproduce
The following sequence illustrates how the problem occurs rather than requiring a deterministic IP-reuse setup:
1. The client caches `server A -> endpoint X`.
2. During a rolling upgrade, server A restarts at endpoint Y, and endpoint X is reused by server B.
3. Before its membership metadata is refreshed, the client creates a connection for server A using endpoint X. The connection is cached under server A's UID, but its actual peer is server B.
4. A later metadata refresh may update `server A -> endpoint Y`, but connection lookup still reuses the existing connection cached by server UID.
5. Requests for buckets led by server A continue to reach server B and receive `NotLeaderOrFollowerException`.
6. Because the exception does not invalidate or disconnect the cached connection, subsequent retries keep using the wrong connection.
### Solution
On `NotLeaderOrFollowerException`, invalidate the affected routing metadata, disconnect the cached connection for the selected leader, refresh cluster metadata, and retry with a bounded retry count.
Also consider replacing an existing connection when the host or port changes for the same server UID.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the client handling for listOffsets requests that receive NotLeaderOrFollowerException, then trace the server-UID connection cache and cluster metadata refresh path. Reproduce the stale endpoint sequence described in the issue and verify that the affected connection and routing metadata are invalidated, metadata is refreshed, retries are bounded, and a changed host or port replaces the cached connection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100