apache / apache/pulsar

improve the accuracy of memory occupation estimate of InMemoryDelayedDeliveryTracker.

Open
#23,699 3 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Now, the memory usage of InMemoryDelayedDeliveryTracker is based on `org.roaringbitmap.longlong.Roaring64Bitmap#getLongSizeInBytes`, referenced in `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker#getBufferMemoryUsage`.

According to the comment of the method,
```
/**
* Estimate of the memory usage of this data structure. This can be expected to be within 1% of
* the true memory usage in common usage scenarios.
* If exact measures are needed, we recommend using dedicated libraries
* such as ehcache-sizeofengine.
*
* In adversarial cases, this estimate may be 10x the actual memory usage. For example, if
* you insert a single random value in a bitmap, then over a 100 bytes may be used by the JVM
* whereas this function may return an estimate of 32 bytes.
*
* The same will be true in the "sparse" scenario where you have a small set of
* random-looking integers spanning a wide range of values.
*
* These are considered adversarial cases because, as a general rule,
* if your data looks like a set
* of random integers, Roaring bitmaps are probably not the right data structure.
*
* Note that you can serialize your Roaring Bitmaps to disk and then construct
* ImmutableRoaringBitmap instances from a ByteBuffer. In such cases, the Java heap
* usage will be significantly less than
* what is reported.
*
* If your main goal is to compress arrays of integers, there are other libraries
* that are maybe more appropriate
* such as JavaFastPFOR.
*
* Note, however, that in general, random integers (as produced by random number
* generators or hash functions) are not compressible.
* Trying to compress random data is an adversarial use case.
*
* @see JavaFastPFOR
*
*
* @return estimated memory usage.
*/
@Override
public long getLongSizeInBytes() {
// 'serializedSizeInBytes' is a better than nothing estimation of the memory footprint
// It would generally be an optimistic estimator (by underestimating the size in memory)
return serializedSizeInBytes();
}
```
We can see that, the essitimate value is within 1% of the true memory usage in common usage scenarios, while in adversarial cases, this estimate may be 10x the actual memory usage, which range wide.

As delayed message queue consume a lot memory, we need to measure it more accurate.

### Solution

According to the comment of method, a lib called `ehcache-sizeofengine` is recommended.
https://github.com/ehcache/sizeof
I wonder whether it is the best solution, is there any lib better than it?

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker#getBufferMemoryUsage and review its use of org.roaringbitmap.longlong.Roaring64Bitmap#getLongSizeInBytes. Read the linked ehcache-sizeofengine project and compare whether it provides a more accurate measurement. Done means an agreed measurement approach and corresponding implementation scope are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.