[PIP-219] Support full scan and trim ledger
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Motivation
Broker uses the `Trimledgers` thread to clean up outdated ledgers. During cleaning, each Broker traverses the topic metadata in memory to find the ledger that reach the retention or TTL threshold.
However, there are some problems with this approach. When a topic has no producer and consumer, Broker deletes the metadata of topic from memory. As a result, ledgers of these topics can never be deleted.
Therefore, we need a way to scan and clean all outdated ledgers .
### Goal
The full scan will cause a large number of requests to the ZooKeeper.
Therefore, the existing cleanup mode will be retained and a full scan mode will be added.
### API Changes
1. Add a new scheduling thread pool
2. Add the following configuration item:
// Full scan interval. This parameter is enabled only when the value > 0.
fullScanTrimLedgerInterval=0
// Maximum number of Metadata requests per second during scanning
fullScanMaximumMetadataConcurrencyPerSecond=200
3. Add a `TrimLedger` admin API.
### Implementation
1. Only the Leader Broker performs full scan.
2. Leader Broker traverse `managedLedger` in each namespace from meta store . Since Ledger metadata contains the creation time. If the creation time is greater than the retention time + TTL time or size, Ledger should be deleted.
Only the metadata of Ledger is parsed instead of loading all topics to the memory.
The metadata request frequency is limited using semaphore.
3. When a topic that meets the conditions, the Leader triggers the `TrimLedger` admin API. Since the admin API will verify the attribution of the topic and redirect it, the topic will be loaded by the corresponding Broker.
After cleaning is done, the corresponding Broker closes the topic to release memory(Before closing, it will check whether the topic has producers and consumers, if not, close it).
### Alternatives
_No response_
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.