apache / apache/pulsar

[PIP-219] Support full scan and trim ledger

Open
#18,128 21 comments 1 reaction 1 assignee Claimed by @315157973 View on GitHub
Stale type/PIP
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Motivation

Broker uses the `Trimledgers` thread to clean up outdated ledgers. During cleaning, each Broker traverses the topic metadata in memory to find the ledger that reach the retention or TTL threshold.
However, there are some problems with this approach. When a topic has no producer and consumer, Broker deletes the metadata of topic from memory. As a result, ledgers of these topics can never be deleted.
Therefore, we need a way to scan and clean all outdated ledgers .

### Goal

The full scan will cause a large number of requests to the ZooKeeper.
Therefore, the existing cleanup mode will be retained and a full scan mode will be added.

### API Changes

1. Add a new scheduling thread pool

2. Add the following configuration item:
// Full scan interval. This parameter is enabled only when the value > 0.
fullScanTrimLedgerInterval=0
// Maximum number of Metadata requests per second during scanning
fullScanMaximumMetadataConcurrencyPerSecond=200

3. Add a `TrimLedger` admin API.

### Implementation

1. Only the Leader Broker performs full scan.
2. Leader Broker traverse `managedLedger` in each namespace from meta store . Since Ledger metadata contains the creation time. If the creation time is greater than the retention time + TTL time or size, Ledger should be deleted.
Only the metadata of Ledger is parsed instead of loading all topics to the memory.
The metadata request frequency is limited using semaphore.

3. When a topic that meets the conditions, the Leader triggers the `TrimLedger` admin API. Since the admin API will verify the attribution of the topic and redirect it, the topic will be loaded by the corresponding Broker.
After cleaning is done, the corresponding Broker closes the topic to release memory(Before closing, it will check whether the topic has producers and consumers, if not, close it).

### Alternatives

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.