apache / apache/bookkeeper

Bookie compaction halts without any errors or diagnostic information.

Open
#4,596 13 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**

***Describe the bug***

Bookie compaction halts without any errors or diagnostic information, even with the DEBUG log level enabled.

We observed that every time it silently stopped, it was at entry log 1884, corresponding to the file 75c.log.

```
# {pod="pulsar-bookie-4",namespace="pulsar"}
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18113 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18120 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-22T09:12:37,547+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.DefaultEntryLogger - Recovering ledgers maps for log 1884 at offset: 282035685
2025-04-22T09:12:38,518+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again.
```

When the entry log 75c.log was moved to a different location and compaction was manually triggered, the bookie successfully reclaimed storage.

Naively checking the consistency of the affected file via:
```/pulsar/bin/bookkeeper shell readlog 1884```
did not reveal any obvious problems with the metadata.

We couldn't find any relevant information at the metrics level either.

The problem occurred again on a different bookie at a different time, with exactly the same symptoms and resolution.

***To Reproduce***

We cannot reliably reproduce the issue. The only noteworthy observation is that both impacted entry log files were closed during a bookie shutdown.

***Expected behavior***

1. Compaction should not stop due to problems with a single file.
2. If compaction halts, it should provide diagnostic information to help identify the issue.
3. There should be a CLI tool or command to check the consistency of an entry log file (if such a tool exists, it may not be well-documented).

***Screenshots***

N/A

***Additional context***

Environment:

- Pulsar Version: 4.0.0
- Apache Bookie Version: 4.17.1
- Deployment:
Deployed via [Pulsar Helm Charts](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html) to Kubernetes with Karpenter (pods may be rescheduled multiple times a day).

Configuration:
```
gcWaitTime: "300000"
isForceGCAllowWhenNoSpace: "true"
majorCompactionInterval: "10800"
majorCompactionThreshold: "0.8"
minorCompactionInterval: "360"
minorCompactionThreshold: "0.2"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.