quickwit-oss / quickwit-oss/quickwit
ExistQuery may not be accurate when relying on fast-field
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
ExistQuery such as field:* can be wrong when they are satisfied from fast-field storage instead of index_field_presence.
When fast field normalizer is set to raw, the fast-field path treats the whole string as one token and drops values longer than 255 bytes, so nothing is written to the fast column for those docs.
Queries like NOT field:* can then exclude documents that do have field in the source JSON, while term queries on the same field (e.g. field:"github") still work because they use the inverted index.
Suggestions:
Maybe instead of dropping the values we could truncate the values, this will break ordering on that fast column but at the moment with the logic of dropping, it's already broken.
Quickwit version
- Quickwit 0.8.2
- Latest dockerhub image (quickwit/quickwit:edge-slim-bookworm)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with a raw fast field containing a value longer than 255 bytes, then compare field:* and NOT field:* with a term query such as field:"github". Start by tracing the fast-field path and its interaction with index_field_presence. Done means existence queries remain accurate while term queries continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100