apache / apache/fluss

[FIP-28] Refresh server partition status for historical write routing

Open
#4,161 0 comments 0 reactions 0 assignees View on GitHub
component=client priority=normal
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Parent issue: #3631

Follow-up to #4119 and PR #4120.

Historical write routing currently uses the client-side auto-partition retention boundary as a precheck before refreshing metadata for an original partition. The client calculation is not authoritative: the client and Coordinator may evaluate time expressions with different default time zones, and the client metadata cache may remain stale while the server creates, retains, freezes, or removes a partition. Performing a synchronous metadata refresh on the per-record path is also too expensive.

Introduce a periodically refreshed, server-authoritative partition status for historical write routing. The client should fetch and cache partition existence or lifecycle information from the server in a batched and rate-limited manner, rather than repeatedly deriving the routing decision from its local clock.

The implementation should:

- Refresh relevant partition status periodically or when server metadata indicates that the cached status is stale, without synchronous RPCs on every record.
- Deduplicate concurrent refreshes per table or original partition path.
- Keep writes on the original partition while the server reports it as active or still present.
- Permit historical routing only after the server reports a state that deterministically allows it. This should align with the durable partition retirement protocol discussed in #3820, where a generic missing or freezing state is not sufficient to reroute.
- Treat client-side time expressions only as an optional refresh hint, not as the source of truth for partition existence or retirement.
- Define cache invalidation and recovery behavior across metadata refreshes and client restarts.

This avoids early routing caused by client/server time-zone differences and reduces late detection caused by stale metadata, while keeping metadata I/O off the per-record hot path.

### Willingness to contribute

- [x] I am willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the client-side auto-partition retention-boundary check and metadata refresh path, then read #4119, PR #4120, and the durable retirement protocol discussed in #3820. Done means server-authoritative partition status is refreshed and cached in batches with rate limiting and deduplication, avoids per-record synchronous RPCs, and defines invalidation and recovery behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.