RamUsageEstimator.sizeOf(Query, long) does not match RamUsageTester.ramUsed(obj)
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
# Context
While implementing `Accountable.ramBytesUsed()`, I noticed a discrepancy between the values returned by `RamUsageEstimator.sizeOf(Query, long)` and `RamUsageTester.ramUsed(obj)` in Lucene 10.2.2.
For example, given the following test:
```
@Test
void testTermQueryRamUsage() {
Query query = new TermQuery(new Term("field", "value"));
long actual = RamUsageTester.ramUsed(query); // 152 bytes
long expected = RamUsageEstimator.sizeOf(query, 0); // 176 bytes
assertThat(actual).isEqualTo(expected);
}
```
`RamUsageTester` reports: **152 bytes**
`RamUsageEstimator` reports: **176 bytes** (using `RamUsageQueryVisitor` internally)
# Observed RamUsageEstimator Flow
1. Invoke `RamUsageEstimator.sizeOf(query, 0)` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L367C22-L367C28))
a) Since TermQuery does not implement the Accountable interface, it should use `RamUsageQueryVisitor`
2. Create an instance of `RamUsageQueryVisitor`(see [constructor](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L308))
a) No default size was passed, so it will invoke `RamUsageEstimator.shallowSizeOf(Query)`, which in my case resulted in **24 bytes**
3. The query uses RamUsageQueryVisitor to traverse through
a) We reach `RamUsageQueryVisitor.consumeTerms` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L319)) . Since query == root, we skip that and go directly into invoking sizeOf(terms)
b) `sizeOf(terms)` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L623)) sets size to `shallowSizeOf(accountables)`, which resulted in **24 bytes**, and then sums together with results of `Accountable.ramBytesUsed()`
c) `Term.ramBytesUsed()` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/index/Term.java#L189)) returns back a total of: 48 + 56 + 24 = **128 bytes**
- `BASE_RAM_BYTES` = **48 bytes** (from shallow size of Term and BytesRef)
- `RamUsageEstimator.sizeOfObject(field)` = **56 bytes**
- `RamUsageEstimator.alignObjectSize(
bytes.bytes.length + RamUsageEstimator.NUM_BYTES_ARRAY_HEADER)` = aligned(5 + 16) = **24 bytes**
4. The total returned is **176 bytes** (24 + 128 + 24), which is **24 bytes more** than the actual usage.
# Problem: Double Counting
In step 3b, the method sets the initial size to the shallow size of the Accountable[] array:
```
public static long sizeOf(Accountable[] accountables) {
long size = shallowSizeOf(accountables); // Term shallow size is 24 bytes
for (Accountable accountable : accountables) {
if (accountable != null) {
size += accountable.ramBytesUsed();
}
}
return size;
}
```
This means the shallow size of the Term is counted twice. Once in `shallowSizeOf(accountables)` and again within each `Accountable.ramBytesUsed()`.
Contributor guide
Research direction
Start in lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java, reading sizeOf(Query, long), RamUsageQueryVisitor.consumeTerms, and sizeOf(Accountable[]); compare their accounting with RamUsageTester.ramUsed(query). Review Term.ramBytesUsed() in lucene/core/src/java/org/apache/lucene/index/Term.java and add a regression test for the TermQuery example. Done means both measurements agree without counting the Term shallow size twice.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100