apache / apache/lucene

RamUsageEstimator.sizeOf(Query, long) does not match RamUsageTester.ramUsed(obj)

Open
#15,026 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

# Context
While implementing `Accountable.ramBytesUsed()`, I noticed a discrepancy between the values returned by `RamUsageEstimator.sizeOf(Query, long)` and `RamUsageTester.ramUsed(obj)` in Lucene 10.2.2.

For example, given the following test:

```
@Test
void testTermQueryRamUsage() {
Query query = new TermQuery(new Term("field", "value"));
long actual = RamUsageTester.ramUsed(query); // 152 bytes
long expected = RamUsageEstimator.sizeOf(query, 0); // 176 bytes
assertThat(actual).isEqualTo(expected);
}
```

`RamUsageTester` reports: **152 bytes**
`RamUsageEstimator` reports: **176 bytes** (using `RamUsageQueryVisitor` internally)

# Observed RamUsageEstimator Flow
1. Invoke `RamUsageEstimator.sizeOf(query, 0)` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L367C22-L367C28))
a) Since TermQuery does not implement the Accountable interface, it should use `RamUsageQueryVisitor`
2. Create an instance of `RamUsageQueryVisitor`(see [constructor](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L308))
a) No default size was passed, so it will invoke `RamUsageEstimator.shallowSizeOf(Query)`, which in my case resulted in **24 bytes**
3. The query uses RamUsageQueryVisitor to traverse through
a) We reach `RamUsageQueryVisitor.consumeTerms` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L319)) . Since query == root, we skip that and go directly into invoking sizeOf(terms)
b) `sizeOf(terms)` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java#L623)) sets size to `shallowSizeOf(accountables)`, which resulted in **24 bytes**, and then sums together with results of `Accountable.ramBytesUsed()`
c) `Term.ramBytesUsed()` (see [method](https://github.com/apache/lucene/blob/279eb7aaafe985e5d0552b7f2a10b63185a3f893/lucene/core/src/java/org/apache/lucene/index/Term.java#L189)) returns back a total of: 48 + 56 + 24 = **128 bytes**
- `BASE_RAM_BYTES` = **48 bytes** (from shallow size of Term and BytesRef)
- `RamUsageEstimator.sizeOfObject(field)` = **56 bytes**
- `RamUsageEstimator.alignObjectSize(
bytes.bytes.length + RamUsageEstimator.NUM_BYTES_ARRAY_HEADER)` = aligned(5 + 16) = **24 bytes**
4. The total returned is **176 bytes** (24 + 128 + 24), which is **24 bytes more** than the actual usage.

# Problem: Double Counting
In step 3b, the method sets the initial size to the shallow size of the Accountable[] array:

```
public static long sizeOf(Accountable[] accountables) {
long size = shallowSizeOf(accountables); // Term shallow size is 24 bytes
for (Accountable accountable : accountables) {
if (accountable != null) {
size += accountable.ramBytesUsed();
}
}
return size;
}
```

This means the shallow size of the Term is counted twice. Once in `shallowSizeOf(accountables)` and again within each `Accountable.ramBytesUsed()`.

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java, reading sizeOf(Query, long), RamUsageQueryVisitor.consumeTerms, and sizeOf(Accountable[]); compare their accounting with RamUsageTester.ramUsed(query). Review Term.ramBytesUsed() in lucene/core/src/java/org/apache/lucene/index/Term.java and add a regression test for the TermQuery example. Done means both measurements agree without counting the Term shallow size twice.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.