quickwit-oss / quickwit-oss/quickwit
No `num_docs` optimization and lazy top K.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
If num_docs is not required, there are many optimization we can run.
For instance, if we sort by docs we can often searhc on only one split and abort search within the split.
If we search by -date too, we can sort splits by order of their max_date, and stop search as soon as we get a guarantee that no docs will enter the top K.
We can hardcode these optimization for the moment, and revisit this if someone has some great formalism to get a proper distributed execution plan abstraction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating the distributed search path that handles optional num_docs, document sorting, and date ordering, then determine how split-level bounds affect top-K results. Done means the proposed early-abort and lazy top-K behavior is implemented without changing result correctness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100