apache / apache/pulsar

The bookkeeper ledger no honoring the retention period and causing for the more disk usage.

Open
#6,639 6 comments 1 reaction 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
The bookkeeper ledger no honoring the retention period and causing for the more disk usage.
During my test execution i set **defaultRetentionTimeInMinutes is 6** and noticed that bookie ledger is no honoring the retention period and eating up more disk space and causing for failures due to "Not enough non-faulty bookie available". When drilled down to disk and see, most of message are retaining even after retention period is completed in ledger disk. The below are the list of broker paramter settins

broker:
configData:
managedLedgerMinLedgerRolloverTimeMinutes: "1"
managedLedgerCursorRolloverTimeInSeconds: "60"
managedLedgerMaxLedgerRolloverTimeMinutes: "2"
managedLedgerMaxEntriesPerLedger: "50000"
defaultRetentionTimeInMinutes: "6"
defaultRetentionSizeInMB: "1000"
ttlDurationDefaultInSeconds: "3600"

**To Reproduce**
Steps to reproduce the behavior:
1. Use 2.5.0 Pulsar
2. Create 100 topics with three partitions and 100 producers and consumers
3. By using pulsar perf client send data for an hour continuously ( message size is 50 kb) and observe whether disk is flushing out or not after completion of the retention period
4. I got the below ERROR message in broker logs
"Not enough non-faulty bookie available"

**Expected behavior**
As per retentionperiod the data should get flushed out and make disk space free

**Screenshots**
n/a

**Desktop (please complete the following information):**
- OS: CentOS 7.7

**Additional context**
Currently we have 100 GB for each bookie
Below is the disk calculation as per my messages/sec.
a=msg/sec -844.023/sec=844.023*50 kb =42201.2 msgs/sec in KB
b=a*retention period = 42201.2*360(6 minutes)=15192414 msg/sec =14.4 GB (3 bookies) is required for 6 minutes duration.
c= 100 GB each bookie (300 GB for 3 bookies)
so b

Contributor guide

Open the contributing guide

Research direction

Reproduce on Pulsar 2.5.0 with the listed managed-ledger and retention settings, using the 100-topic workload and pulsar perf client. Start by tracing retention handling from defaultRetentionTimeInMinutes through ledger cleanup and bookie disk usage; done means data is released after the retention period and the "Not enough non-faulty bookie available" error no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.