apache / apache/pulsar

Topic partition's segmented storage space calculation error

Open
#12,399 6 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
The storage space of partition is much larger than 4G and entries is much larger than 50000 (managedLedgerMaxSizePerLedgerMbytes=4096, managedLedgerMaxEntriesPerLedger=50000 in broker.conf).

Pulsar v2.8.1
![e898c53b8dbd468fb4d0ed0de0a2c2a](https://user-images.githubusercontent.com/54351417/137667693-64cc5156-77b4-427d-bf7a-789cdd32f35a.png)

broker.conf
managedLedgerMaxEntriesPerLedger=50000
managedLedgerMaxSizePerLedgerMbytes=4096

```
producer code:
producer = client.newProducer(Schema.STRING)
.topic(topic)
.producerName(producerName)
.sendTimeout(0, TimeUnit.SECONDS)
.enableBatching(true)
.batchingMaxMessages(1000)
.compressionType(CompressionType.ZSTD)
.blockIfQueueFull(true)
.create();
```

**To Reproduce**
Steps to reproduce the behavior:
1. Start producer
2. View the segmented storage of the partition in pulsar manager
3. The storage space of partition is much larger than 4G

Contributor guide

Open the contributing guide

Research direction

Start with the broker.conf limits and the Java producer settings, then reproduce the partition in Pulsar Manager as described. Compare the displayed segmented storage and entry counts with the configured limits; done means the partition no longer exceeds the expected 4G or 50,000-entry thresholds and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.