Bookie compaction halts without any errors or diagnostic information.
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
**BUG REPORT**
***Describe the bug***
Bookie compaction halts without any errors or diagnostic information, even with the DEBUG log level enabled.
We observed that every time it silently stopped, it was at entry log 1884, corresponding to the file 75c.log.
```
# {pod="pulsar-bookie-4",namespace="pulsar"}
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18113 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18120 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-22T09:12:37,547+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.DefaultEntryLogger - Recovering ledgers maps for log 1884 at offset: 282035685
2025-04-22T09:12:38,518+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again.
```
When the entry log 75c.log was moved to a different location and compaction was manually triggered, the bookie successfully reclaimed storage.
Naively checking the consistency of the affected file via:
```/pulsar/bin/bookkeeper shell readlog 1884```
did not reveal any obvious problems with the metadata.
We couldn't find any relevant information at the metrics level either.
The problem occurred again on a different bookie at a different time, with exactly the same symptoms and resolution.
***To Reproduce***
We cannot reliably reproduce the issue. The only noteworthy observation is that both impacted entry log files were closed during a bookie shutdown.
***Expected behavior***
1. Compaction should not stop due to problems with a single file.
2. If compaction halts, it should provide diagnostic information to help identify the issue.
3. There should be a CLI tool or command to check the consistency of an entry log file (if such a tool exists, it may not be well-documented).
***Screenshots***
N/A
***Additional context***
Environment:
- Pulsar Version: 4.0.0
- Apache Bookie Version: 4.17.1
- Deployment:
Deployed via [Pulsar Helm Charts](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html) to Kubernetes with Karpenter (pods may be rescheduled multiple times a day).
Configuration:
```
gcWaitTime: "300000"
isForceGCAllowWhenNoSpace: "true"
majorCompactionInterval: "10800"
majorCompactionThreshold: "0.8"
minorCompactionInterval: "360"
minorCompactionThreshold: "0.2"
```
Contributor guide
Assessment
This issue has not been assessed yet.