apache / apache/lucene

lengthsBuf in CompressingTermVectorsWriter is a bit redundant [LUCENE-9101]

Open
#10,143 0 comments 0 reactions 0 assignees View on GitHub
affects-version:8.2 legacy-jira-priority:Major module:core/codecs type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In CompressingTermVectorsWriter, We use lengthsBuf to save the length of every terms, for example: "a a a b1 b1 b1 b1", the lengthsBuf=[1,1,1,2,2,2,2], a appears three times, we count three times, it seems a bit redundant.

We use it in CompressingTermVectorsWriter.flushOffsets:

 

```java
private void flushOffsets(int[] fieldNums) throws IOException {
......
// lengths
writer.reset(vectorsStream);
for (DocData dd : pendingDocs) {
for (FieldData fd : dd.fields) {
if ((fd.flags & OFFSETS) != 0) {
int pos = 0;
for (int i = 0; i < fd.numTerms; ++i) {
for (int j = 0; j < fd.freqs[i]; ++j) {
writer.add(lengthsBuf[fd.offStart + pos++] - fd.prefixLengths[i] - fd.suffixLengths[i]);
}
}
assert pos == fd.totalPositions;
}
}
}
writer.finish();
}
```

 

we can simply it: lengthsBuf=[1,2], the same term just count one time. we could use `int count;` to count the current term we are process, for example:

 

```java
private void flushOffsets(int[] fieldNums) throws IOException {
......
// lengths
writer.reset(vectorsStream);
for (DocData dd : pendingDocs) {
for (FieldData fd : dd.fields) {
if ((fd.flags & OFFSETS) != 0) {
int pos = 0;
for (int i = 0; i < fd.numTerms; ++i) {
count ++;
for (int j = 0; j < fd.freqs[i]; ++j) {
writer.add(lengthsBuf[count] - fd.prefixLengths[i] - fd.suffixLengths[i]);
}
}
assert pos == fd.totalPositions;
}
}
}
writer.finish();
}
```

---
Migrated from [LUCENE-9101](https://issues.apache.org/jira/browse/LUCENE-9101) by kkewwei (@kkewwei)

Contributor guide

Open the contributing guide

Research direction

Start at CompressingTermVectorsWriter.flushOffsets and trace how lengthsBuf, prefixLengths, suffixLengths, frequencies, and totalPositions are populated and consumed. Verify the proposed deduplication preserves offset lengths for repeated terms and check the relevant term-vector tests; done means redundant per-occurrence lengths are removed without changing encoded offsets or test behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.