[Feature] Support ManagedLedger EntryCache cache entires before the slowest cursor
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Is your feature request related to a problem? Please describe.**
A topic may have lots of subscriptions in some scenes like an advertising recommendation system. This kind of system may update very frequently and at which point all subscriptions need to seek back to a fixed time(one or two hours).
We have a pulsar cluster for a recommendation business, every topic has 200~400 subscriptions, and all subscriptions need to seek back for 2 hours when the recommendation system restart(update or upgrade). And we want to catch up on all backlogs ASAP.
I.e. all subscriptions are consuming data in tailing read way in peacetime, and if the user's system restarts, all subscriptions will seek to 2 hours ago then read all backlogs and continue consuming new data.
But even after we turning both the `managedLedgerCacheEvictionTimeThresholdMillis` and `managedLedgerCursorBackloggedThreshold` to a very large value, the EntryCache still missed after `consumer.seek()`. This is caused by the `cacheEvictionTask`, which will always remove all entries already read by active cursors.
```java
void doCacheEviction(long maxTimestamp) {
// Always remove all entries already read by active cursors
PositionImpl slowestReaderPos = getEarlierReadPositionForActiveCursors();
if (slowestReaderPos != null) {
entryCache.invalidateEntries(slowestReaderPos);
}
// Remove entries older than the cutoff threshold
entryCache.invalidateEntriesBeforeTimestamp(maxTimestamp);
}
```
**Describe the solution you'd like**
We can add a new configuration `managedLedgerCacheEvictionSkipSlowestCursor` to control whether the `cacheEvictionTask` will remove entries read by all active cursors.
And by default, the `managedLedgerCacheEvictionSkipSlowestCursor` is false, which will keep compatible with logic before. But for scenes that need to read backlog from EntryCachel, the user can set `managedLedgerCacheEvictionSkipSlowestCursor` to true which will cache all entries until reach the `managedLedgerCacheEvictionTimeThresholdMillis`.
Contributor guide
Research direction
Start by locating ManagedLedger's doCacheEviction/cacheEvictionTask and the definitions for managedLedgerCacheEvictionTimeThresholdMillis and managedLedgerCursorBackloggedThreshold. Trace how the proposed managedLedgerCacheEvictionSkipSlowestCursor setting affects active-cursor invalidation, then verify that false preserves current eviction and true retains entries until the time-threshold eviction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100