apache / apache/lucene

FST doesn't fully share common prefix across all outputs [LUCENE-3297]

Open
#4,370 6 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:core/FSTs type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

FST will try to share prefixes of outputs when possible, however in the [I think unusual in practice] case where all outputs share a common prefix, FST really ought to store this just once, on the root arc, but instead it's only able to push back to the N root arcs. It's sort of an off-by-one on how far back the pushing goes...

One [synthetic] example where this makes a big difference is the new Test2BPostings test, when it uses MemoryCodec, because this test has 26 terms (letters of alphabet) and each term has exactly the same long (\~85 MB) all 1s byte[] as the postings. If we fixed this issue, then the resulting FST would only be \~85 MB but now instead it needs to be \~85 \* 26 MB.

---
Migrated from [LUCENE-3297](https://issues.apache.org/jira/browse/LUCENE-3297) by Michael McCandless (@mikemccand), updated Aug 09 2011

Contributor guide

Open the contributing guide

Research direction

Start with the FST implementation and the Test2BPostings case that uses MemoryCodec, then reproduce the synthetic 26-term example with identical large byte outputs. Trace where output prefix pushing stops at the root arcs; done means the common prefix is stored once on the root arc and the test's FST memory use is reduced without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.