[Perf] RemoteHoodieTableFileSystemView triggers repeated server-side timeline reload whenever any new instant appears after the view is constructed
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Describe the problem
When a client uses `RemoteHoodieTableFileSystemView` and any new instant is committed on the timeline **after** the view has been constructed (for example an async operation such as clean, compaction, clustering, or rollback running in a different actor), every subsequent request from that view triggers a full server-side `SyncableFileSystemView#sync()` — i.e. `metaClient.reloadActiveTimeline() + clear() + init()`. The client's cached timeline never updates and the server's timeline is already up-to-date, so the mismatch never converges: the re-sync happens on **every single request** for the entire lifetime of the view instance.
The impact is amplified when a client scans many partitions, e.g. calling `getLatestFileSlice(partition)` across N partitions. Each partition results in one HTTP request, and each request triggers one full timeline reload on the server. N partitions → N wasted syncs.
## Root cause
Two pieces of code combine to produce the pathology:
**(1) The client's local timeline is a constructor-time snapshot and is never refreshed.**
`RemoteHoodieTableFileSystemView` assigns `this.timeline` exactly once in the constructor:
```java
if (metaClient.getTableConfig().isLSMBasedLogFormat()) {
this.timeline = metaClient.getActiveTimeline().filterCompletedAndReplaceInstants();
} else {
this.timeline = metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
}
```
There is no code path that refreshes `this.timeline` for the lifetime of the view instance. `executeRequest` then sends `LAST_INSTANT_TS` and `TIMELINE_HASH` derived from that frozen snapshot on every request. As soon as any instant is added to the timeline out-of-band after the view is constructed, the client's hash is permanently stale for that view instance.
**(2) The server treats any hash mismatch as "needs sync".**
`RequestHandler#isLocalViewBehind`:
```java
private boolean isLocalViewBehind(Context ctx) {
...
String localTimelineHash = localTimeline.getTimelineHash();
// refresh if timeline hash mismatches
if (!localTimelineHash.equals(timelineHashFromClient)) {
return true; // ← ANY mismatch is treated as "server behind"
}
// As a safety check, even if hash is same, ensure instant is present
return !localTimeline.containsOrBeforeTimelineStarts(lastKnownInstantFromClient);
}
```
Judging by the method name, the check is supposed to answer "is the server's view behind what the client saw?". But the implementation collapses two very different situations into the same `true` return:
- **server behind client** → `sync()` is legitimate
- **client behind server** → `sync()` is unnecessary; the server is already newer
The follow-up `containsOrBeforeTimelineStarts(lastKnownInstantFromClient)` check would actually cover the "client behind server" case correctly (the client's last instant is already present in the server timeline, so nothing needs to be re-synced), but it is short-circuited by the earlier `return true` on hash mismatch.
Combined effect: (1) makes the client's hash permanently stale after any out-of-band instant, and (2) turns that stale hash into a per-request full timeline reload on the server.
## Concrete evidence from production logs
In our environment the trigger was an async clean instant being committed after the client's view was constructed, but any new instant produces the same behavior.
Timeline files on HDFS at the time of the incident (last two instants):
```
20260721213743765.deltacommit (completed — this is what the client saw)
20260721213743765.deltacommit.inflight
20260721213743765.deltacommit.requested
20260721213744648.clean (completed — appeared AFTER the client opened its view)
20260721213744648.clean.inflight
20260721213744648.clean.requested
```
A Spark reader then queries many partitions in parallel. The client's `TIMELINE_HASH` is stable (`6bbdf6f1...162bf`), the client's `LAST_INSTANT_TS` is stable (`20260721213743765`, the deltacommit — the client has not observed the newer instant), and the server's view already reflects the newer state. Every single request triggers a "Syncing view..." log on the server:
```
2026-07-21 21:42:47.413 INFO [ForkJoinPool.commonP:l-worker-3] o.a.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://xxx/v1/hoodie/view/slices/partition/latest/?partition=dt%3D2017-02-03&basepath=hdfs%3A%2F%2Fxxxxx&lastinstantts=20260721213743765&timelinehash=6bbdf6f1ea5484010b2275be7e8f1fe993033321bbe8cff9675044b654f162bf)
2026-07-21 21:42:47.575 INFO [qtp2089656019-535557:19-5355571] org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.587 INFO [qtp2089656019-535557:19-5355571] org.apache.hudi.common.table.HoodieTableMetaClient - Skip backtrack instance : []
2026-07-21 21:42:47.587 INFO [qtp2089656019-535557:19-5355571] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 6 ms to read 0 instants, 0 replaced file groups
2026-07-21 21:42:47.612 INFO [qtp2089656019-535557:19-5355571] org.apache.hudi.common.util.ClusteringUtils - Found 0 files in pending clustering operations
2026-07-21 21:42:47.612 INFO [qtp2089656019-535557:19-5355571] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (dt=2014-06-10)
2026-07-21 21:42:47.617 INFO [qtp2089656019-535572:19-5355722] org.apache.hudi.timeline.service.RequestHandler - Syncing view as client passed last known instant 20260721213743765 as last known instant but server has the following last instant on timeline :Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.626 INFO [qtp2089656019-535557:19-5355571] o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in partition (dt=2014-06-10) =11, Time taken =14
2026-07-21 21:42:47.627 INFO [qtp2089656019-535557:19-5355571] o.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=11, NumFileGroups=10, FileGroupsCreationTime=1, StoreTimeTaken=0
2026-07-21 21:42:47.627 INFO [qtp2089656019-535557:19-5355571] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Time to load partition (dt=2014-06-10) =15
2026-07-21 21:42:47.638 INFO [ForkJoinPool.commonP:l-worker-2] o.a.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://xxx/v1/hoodie/view/slices/partition/latest/?partition=dt%3D2014-06-11&basepath=hdfs%3A%2F%2Fxxxxx&lastinstantts=20260721213743765&timelinehash=6bbdf6f1ea5484010b2275be7e8f1fe993033321bbe8cff9675044b654f162bf)
2026-07-21 21:42:47.789 INFO [qtp2089656019-535572:19-5355722] org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.804 INFO [qtp2089656019-535572:19-5355722] org.apache.hudi.common.table.HoodieTableMetaClient - Skip backtrack instance : []
2026-07-21 21:42:47.804 INFO [qtp2089656019-535572:19-5355722] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 8 ms to read 0 instants, 0 replaced file groups
2026-07-21 21:42:47.841 INFO [qtp2089656019-535572:19-5355722] org.apache.hudi.common.util.ClusteringUtils - Found 0 files in pending clustering operations
2026-07-21 21:42:47.841 INFO [qtp2089656019-535572:19-5355722] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (dt=2021-05-19)
2026-07-21 21:42:47.846 INFO [qtp2089656019-535586:19-5355867] org.apache.hudi.timeline.service.RequestHandler - Syncing view as client passed last known instant 20260721213743765 as last known instant but server has the following last instant on timeline :Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722] o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in partition (dt=2021-05-19) =11, Time taken =14
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722] o.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=11, NumFileGroups=10, FileGroupsCreationTime=0, StoreTimeTaken=0
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722] o.apache.hudi.common.table.view.AbstractTableFileSystemView - Time to load partition (dt=2021-05-19) =14
```
## Steps to reproduce
1. Have a Hudi 0.13.1 table (COW or MOR) served by an embedded/standalone timeline server.
2. Open a long-lived `RemoteHoodieTableFileSystemView` from a client (Spark reader, or a Flink component that reads via the remote view).
3. From a different actor (e.g. Flink's async cleaner, an inline clean/compaction, `hudi-cli`, or any writer commit) let a new instant become the newest one on the timeline **after** step 2.
4. From the client opened in step 2, call any partition-level API repeatedly, ideally across many partitions in one query (e.g. `getLatestFileSlice(partition)` for N partitions).
5. Enable INFO logs on `RequestHandler` and `AbstractTableFileSystemView`. You will see:
- the client's `TIMELINE_HASH` in the request URL stays constant across all requests, and
- the server logs a `Syncing view ...` line and a `HoodieActiveTimeline: Loaded instants upto ...` reload on **every** request.
## Environment Description
- **Hudi version**: 0.13.1
- **Affected engines**: both Spark and Flink (both use `RemoteHoodieTableFileSystemView`)
- **Storage**: HDFS
- **Running on Docker?**: No
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing RemoteHoodieTableFileSystemView#executeRequest and RequestHandler#isLocalViewBehind, focusing on the client timeline snapshot, TIMELINE_HASH, LAST_INSTANT_TS, and the server sync decision. Reproduce with a new instant added after view construction and repeated partition-level requests; done means the server does not reload its timeline on every request when it is already newer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100