[Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Bookie Version:4.14.4
Broker Version: 2.9.2
OS: Linux Centos7
### Minimal reproduce step
Under the above version, we found that some EntryLogs in Bookie could not be deleted for a long time. At first, we thought that it was caused by not triggering minor GC or major GC. Then we adjusted the thresholds of minor GC and major GC, and found that the recovery effect was not very obvious. Then we scanned the problematic EntryLog file, and then obtained the list of all Ledgers in the current EntryLog, and obtained the ledger metadata corresponding to each Ledger, and then parsed out the Topic information corresponding to the Ledger, and found the following situation:
1. The ledger metadata as follows:
```
2022-12-26 20:28:06,184 - INFO - [main:ParseEntryLog@217] - ManagedLedger: LedgerID: 1174539 --- LedgerCreateTime: 1654146303273 --- LedgerState: CLOSED --- Ensembles: {0=[xxx:3181, xxx:3181, xxx:3181]} --- TopicName: pulsar-xxx/test/test2-partition-2
```
2. The topic stats internal as follows
[stats-internal.log](https://github.com/apache/pulsar/files/10307780/internal.log)

We can see that the `markDeletePosition` of all subscriptions under the current topic has been updated to the last message, indicating that all the messages in this topic have been correctly consumed and confirmed. But in fact, this ledger has not been deleted, which leads to the fact that the proportion of valid data in the EntryLog does not meet the triggering conditions of major GC, which further causes the EntryLog to exist for a long time and cannot be deleted.
Observing the topic stats internal, we can see that it is currently at a boundary position, markDeletePosition is at the position of the last message of the previous Ledger, and there is no message in the next new Ledger, so whether there is a Ledger in the boundary case If it cannot be deleted, observe that the state of the ledger is already in the CLOSED state
The following is the data in the EntryLog file scanned by the scan script:
[entry.log](https://github.com/apache/pulsar/files/10307846/entry.log)
### What did you expect to see?
The CLOSED ledger will be deleted.
### What did you see instead?
The CLOSED ledger not be deleted.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the minimal reproduction details and the attached stats-internal.log and entry.log files, then trace how markDeletePosition at a ledger boundary affects deletion of CLOSED ledgers. Done means the corresponding CLOSED ledger is deleted in the reported boundary case and the behavior is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100