apache / apache/bookkeeper

The index files under the "locations" folder cannot be completely deleted.

Open
#4,145 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**
The index files with the .sst extension under the "locations" folder cannot be completely deleted.

***Describe the bug***

After upgrading Pulsar to version 3.0.1, I encountered an issue where the index files with the .sst files under the 'locations' folder in the Bookie directory cannot be completely deleted. In clusters with a small data volume, the behavior seems normal, with only a few .sst files for the current day. However, in clusters with high message throughput, such as 20,000 messages per second, I observe that files from several days ago are not being deleted. For instance, one of our clusters accumulated 40 GB of .sst index files over 20 days.After using the command 'bin/bookkeeper shell rebuild-db-ledger-locations-index' to rebuild the index, the overall space occupancy has reduced to 200 MB.

199M locations
46G locations.BACKUP-2023-12-06T03:26:40.383+0000

When examining the 'LOG' files, I found deletion records for some other files, but many files are still not getting deleted. The issue seems to be more pronounced in clusters with a large message volume.

2023/12/06-07:42:27.295286 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030778.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/12/06-07:42:27.296158 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030776.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/12/06-07:42:27.297293 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030774.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000

Could you please provide guidance or assistance on resolving this issue?

***To Reproduce***

Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

***Expected behavior***

A clear and concise description of what you expected to happen.

***Screenshots***

If applicable, add screenshots to help explain your problem.

***Additional context***

Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.