quickwit-oss / quickwit-oss/quickwit
Allow multi indexes search with different query ASTs between indexes
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
In #3734, we introduced the multi indexes search feature.
With one limitation: it works only if the resolved query AST is the same on all indexes. And query AST can be different when indexes have different search default values.
There is a use case where searching on indexes with different default values is beneficial. For example, if you want to search on one index containing logs, another containing transactions, etc. Searching for a specific ID in all the indexes is impossible currently (except if you are searching in a JSON field). But if we allow default_fields in the different indexes will make the search experience possible.
Possible Solution
It's easy to keep around one query AST per index and attach the right query AST to the leaf requests.
The main issue comes from how we list the relevant splits for a given query. In the current code base, we call list_splitswith a ListSplitsQuery argument. Everything works well except for the TagAstFilter as it accepts only one tag AST. However, different default values between indexes will lead to different TagAstFilter.
To avoid that, we could fire multiple list_splits for indexes with different TagAstFilter, which will solve our problem.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the multi-index search flow from list_splits and its ListSplitsQuery argument through TagAstFilter to the leaf requests. Compare how query ASTs and default values are resolved for each index. Done means indexes with different default values can search the same request, including ID searches, without relying on JSON fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100