apache / apache/pulsar

maxBatchSize preallocated memory may be thousands of times larger than actual message length

Open
#14,943 6 comments 0 reactions 1 assignee Claimed by @tjiuming View on GitHub
lifecycle/stale Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your enhancement request related to a problem? Please describe.**
Guys, during the stress test, we found that the client's memory is very large, and even Full gc appeared. After analyzing the dump file, we found that the memory space occupied is much larger than the actual message size, and the serialization of 1KB of messages takes up 1MB.
```
batchedMessageMetadataAndPayload = PulsarByteBufAllocator.DEFAULT
.buffer(Math.min(maxBatchSize, ClientCnx.getMaxMessageSize()));
```

`maxBatchSize = Math.max(maxBatchSize, uncompressedSize);`

Debugging found that the maxBatchSize that controls the pre-allocated `ByteBufPair.b2` memory size is stateful. As the size of the largest batch or the largest single message grows, this may cause the pre-allocated `ByteBufPair.b2` memory to grow larger and larger , which maybe thousands of times larger than the payload of MessageImpl.

Lowering the value of `batchingMaxMessages `may reduce the risk of problems, but a single message that may be too large can also cause problems

**Describe the solution you'd like**
It is best to loop through the messages to be packed to accurately calculate the memory size to be allocated

**Describe alternatives you've considered**
Let the user choose whether to precisely allocate or pre-allocate

**Additional context**
```
maxPendingMessages=2000
maxPendingMessagesAcrossPartitions=40000
blockIfQueueFull=false
sendTimeoutMs=5000
batchingMaxPublishDelayMicros=50
batchingMaxMessages=2000
batchingMaxBytes=5242880
batchingEnabled=true
```

![A11E7121-0FC6-4A0F-AC88-FC33AF63F41C](https://user-images.githubusercontent.com/9923278/160776933-0dadf49b-8daf-40f2-ac36-d11857d7f3d8.jpeg)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.