[FIP-28] Refresh server partition status for historical write routing
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Parent issue: #3631
Follow-up to #4119 and PR #4120.
Historical write routing currently uses the client-side auto-partition retention boundary as a precheck before refreshing metadata for an original partition. The client calculation is not authoritative: the client and Coordinator may evaluate time expressions with different default time zones, and the client metadata cache may remain stale while the server creates, retains, freezes, or removes a partition. Performing a synchronous metadata refresh on the per-record path is also too expensive.
Introduce a periodically refreshed, server-authoritative partition status for historical write routing. The client should fetch and cache partition existence or lifecycle information from the server in a batched and rate-limited manner, rather than repeatedly deriving the routing decision from its local clock.
The implementation should:
- Refresh relevant partition status periodically or when server metadata indicates that the cached status is stale, without synchronous RPCs on every record.
- Deduplicate concurrent refreshes per table or original partition path.
- Keep writes on the original partition while the server reports it as active or still present.
- Permit historical routing only after the server reports a state that deterministically allows it. This should align with the durable partition retirement protocol discussed in #3820, where a generic missing or freezing state is not sufficient to reroute.
- Treat client-side time expressions only as an optional refresh hint, not as the source of truth for partition existence or retirement.
- Define cache invalidation and recovery behavior across metadata refreshes and client restarts.
This avoids early routing caused by client/server time-zone differences and reduces late detection caused by stale metadata, while keeping metadata I/O off the per-record hot path.
### Willingness to contribute
- [x] I am willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the client-side auto-partition retention-boundary check and metadata refresh path, then read #4119, PR #4120, and the durable retirement protocol discussed in #3820. Done means server-authoritative partition status is refreshed and cached in batches with rate limiting and deduplication, avoids per-record synchronous RPCs, and defines invalidation and recovery behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100