apache / apache/pulsar

[Bug] [ml] Incorrect behavior of Topic Retention Policy

Open
#22,473 9 comments 0 reactions 1 assignee Claimed by @dao-jun View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

master branch

### Minimal reproduce step

```java
public void testRetentionInMB() throws Exception {
@Cleanup("shutdown")
ManagedLedgerFactory factory = new ManagedLedgerFactoryImpl(metadataStore, bkc);
ManagedLedgerConfig config = new ManagedLedgerConfig();
config.setRetentionSizeInMB(2);
config.setRetentionTime(-1, TimeUnit.SECONDS);
config.setMaxEntriesPerLedger(2);
config.setMinimumRolloverTime(0, TimeUnit.SECONDS);

@Cleanup
ManagedLedgerImpl ml = (ManagedLedgerImpl) factory.open("retention_test_ledger", config);
@Cleanup
ManagedCursor c1 = ml.openCursor("c1");

// 512K per entry, 2 entries per ledger, 1 ledger has 1MB data.
byte[] content = new byte[1024 * 512];
for (int i = 0; i < 21; i++) {
ml.addEntry(content);
}

// should be 11 ledgers.
assertEquals(ml.ledgers.size(), (21 / 2) + 1);
long firstKey = ml.ledgers.firstKey();

// ack 5 ledgers.
c1.markDelete(new PositionImpl(firstKey + 5, 0));

// trigger trim ledgers manually
CompletableFuture f = new CompletableFuture<>();
ml.trimConsumedLedgersInBackground(false, f);
f.get();
}
```

1. For the test, we set the retention policy by size(2MB); 2 entries per ledger and each entry 512KB, each ledger is 1MB.
2. Add 21 entries to the ManagedLedger, it will open 11 ledgers(I assume these ledger ID are **[1, 11]**).
3. Set markDeletePosition to **6:0**, which means **[1, 5]** ledgers are consumed
4. Trigger trim ledger task manually
5. Wait until trim task finish

### What did you expect to see?

Ledgers **[1, 3]** deleted.

### What did you see instead?

Ledgers **[1, 5]** deleted

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.