lengthsBuf in CompressingTermVectorsWriter is a bit redundant [LUCENE-9101]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In CompressingTermVectorsWriter, We use lengthsBuf to save the length of every terms, for example: "a a a b1 b1 b1 b1", the lengthsBuf=[1,1,1,2,2,2,2], a appears three times, we count three times, it seems a bit redundant.
We use it in CompressingTermVectorsWriter.flushOffsets:
```java
private void flushOffsets(int[] fieldNums) throws IOException {
......
// lengths
writer.reset(vectorsStream);
for (DocData dd : pendingDocs) {
for (FieldData fd : dd.fields) {
if ((fd.flags & OFFSETS) != 0) {
int pos = 0;
for (int i = 0; i < fd.numTerms; ++i) {
for (int j = 0; j < fd.freqs[i]; ++j) {
writer.add(lengthsBuf[fd.offStart + pos++] - fd.prefixLengths[i] - fd.suffixLengths[i]);
}
}
assert pos == fd.totalPositions;
}
}
}
writer.finish();
}
```
we can simply it: lengthsBuf=[1,2], the same term just count one time. we could use `int count;` to count the current term we are process, for example:
```java
private void flushOffsets(int[] fieldNums) throws IOException {
......
// lengths
writer.reset(vectorsStream);
for (DocData dd : pendingDocs) {
for (FieldData fd : dd.fields) {
if ((fd.flags & OFFSETS) != 0) {
int pos = 0;
for (int i = 0; i < fd.numTerms; ++i) {
count ++;
for (int j = 0; j < fd.freqs[i]; ++j) {
writer.add(lengthsBuf[count] - fd.prefixLengths[i] - fd.suffixLengths[i]);
}
}
assert pos == fd.totalPositions;
}
}
}
writer.finish();
}
```
---
Migrated from [LUCENE-9101](https://issues.apache.org/jira/browse/LUCENE-9101) by kkewwei (@kkewwei)
Contributor guide
Research direction
Start at CompressingTermVectorsWriter.flushOffsets and trace how lengthsBuf, prefixLengths, suffixLengths, frequencies, and totalPositions are populated and consumed. Verify the proposed deduplication preserves offset lengths for repeated terms and check the relevant term-vector tests; done means redundant per-occurrence lengths are removed without changing encoded offsets or test behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100