apache / apache/fluss

[client] NotLeaderOrFollowerException persists when a stale TabletServer endpoint is reused

Open
#4,256 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.8.0

### Please describe the bug 🐞

During a rolling upgrade, a TabletServer may restart with a new endpoint, while its old endpoint is reused by another TabletServer. A client may then keep a cached connection for the original server ID that actually connects to the other TabletServer.

A request such as `listOffsets` reaches the wrong server and receives `NotLeaderOrFollowerException`. The current handling completes the future exceptionally without disconnecting the cached connection or forcing a cluster metadata refresh. Subsequent caller-side retries reuse the same connection, making the failure persistent.

This is related to #2097 and #3389. Unlike the transient leader-change case described in #3389, the stale endpoint remains reachable because it now belongs to another TabletServer, so no `NetworkException` is raised to invalidate the connection.

### Steps to reproduce

The following sequence illustrates how the problem occurs rather than requiring a deterministic IP-reuse setup:

1. The client caches `server A -> endpoint X`.
2. During a rolling upgrade, server A restarts at endpoint Y, and endpoint X is reused by server B.
3. Before its membership metadata is refreshed, the client creates a connection for server A using endpoint X. The connection is cached under server A's UID, but its actual peer is server B.
4. A later metadata refresh may update `server A -> endpoint Y`, but connection lookup still reuses the existing connection cached by server UID.
5. Requests for buckets led by server A continue to reach server B and receive `NotLeaderOrFollowerException`.
6. Because the exception does not invalidate or disconnect the cached connection, subsequent retries keep using the wrong connection.

### Solution

On `NotLeaderOrFollowerException`, invalidate the affected routing metadata, disconnect the cached connection for the selected leader, refresh cluster metadata, and retry with a bounded retry count.

Also consider replacing an existing connection when the host or port changes for the same server UID.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the client handling for listOffsets requests that receive NotLeaderOrFollowerException, then trace the server-UID connection cache and cluster metadata refresh path. Reproduce the stale endpoint sequence described in the issue and verify that the affected connection and routing metadata are invalidated, metadata is refreshed, retries are bounded, and a changed host or port replaces the cached connection.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.