improve the accuracy of memory occupation estimate of InMemoryDelayedDeliveryTracker.
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Now, the memory usage of InMemoryDelayedDeliveryTracker is based on `org.roaringbitmap.longlong.Roaring64Bitmap#getLongSizeInBytes`, referenced in `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker#getBufferMemoryUsage`.
According to the comment of the method,
```
/**
* Estimate of the memory usage of this data structure. This can be expected to be within 1% of
* the true memory usage in common usage scenarios.
* If exact measures are needed, we recommend using dedicated libraries
* such as ehcache-sizeofengine.
*
* In adversarial cases, this estimate may be 10x the actual memory usage. For example, if
* you insert a single random value in a bitmap, then over a 100 bytes may be used by the JVM
* whereas this function may return an estimate of 32 bytes.
*
* The same will be true in the "sparse" scenario where you have a small set of
* random-looking integers spanning a wide range of values.
*
* These are considered adversarial cases because, as a general rule,
* if your data looks like a set
* of random integers, Roaring bitmaps are probably not the right data structure.
*
* Note that you can serialize your Roaring Bitmaps to disk and then construct
* ImmutableRoaringBitmap instances from a ByteBuffer. In such cases, the Java heap
* usage will be significantly less than
* what is reported.
*
* If your main goal is to compress arrays of integers, there are other libraries
* that are maybe more appropriate
* such as JavaFastPFOR.
*
* Note, however, that in general, random integers (as produced by random number
* generators or hash functions) are not compressible.
* Trying to compress random data is an adversarial use case.
*
* @see JavaFastPFOR
*
*
* @return estimated memory usage.
*/
@Override
public long getLongSizeInBytes() {
// 'serializedSizeInBytes' is a better than nothing estimation of the memory footprint
// It would generally be an optimistic estimator (by underestimating the size in memory)
return serializedSizeInBytes();
}
```
We can see that, the essitimate value is within 1% of the true memory usage in common usage scenarios, while in adversarial cases, this estimate may be 10x the actual memory usage, which range wide.
As delayed message queue consume a lot memory, we need to measure it more accurate.
### Solution
According to the comment of method, a lib called `ehcache-sizeofengine` is recommended.
https://github.com/ehcache/sizeof
I wonder whether it is the best solution, is there any lib better than it?
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start at org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker#getBufferMemoryUsage and review its use of org.roaringbitmap.longlong.Roaring64Bitmap#getLongSizeInBytes. Read the linked ehcache-sizeofengine project and compare whether it provides a more accurate measurement. Done means an agreed measurement approach and corresponding implementation scope are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100