quickwit-oss / quickwit-oss/quickwit

Allow multi indexes search with different query ASTs between indexes

Open
#3,755 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

In #3734, we introduced the multi indexes search feature.

With one limitation: it works only if the resolved query AST is the same on all indexes. And query AST can be different when indexes have different search default values.

There is a use case where searching on indexes with different default values is beneficial. For example, if you want to search on one index containing logs, another containing transactions, etc. Searching for a specific ID in all the indexes is impossible currently (except if you are searching in a JSON field). But if we allow default_fields in the different indexes will make the search experience possible.

Possible Solution

It's easy to keep around one query AST per index and attach the right query AST to the leaf requests.

The main issue comes from how we list the relevant splits for a given query. In the current code base, we call list_splitswith a ListSplitsQuery argument. Everything works well except for the TagAstFilter as it accepts only one tag AST. However, different default values between indexes will lead to different TagAstFilter.

To avoid that, we could fire multiple list_splits for indexes with different TagAstFilter, which will solve our problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the multi-index search flow from list_splits and its ListSplitsQuery argument through TagAstFilter to the leaf requests. Compare how query ASTs and default values are resolved for each index. Done means indexes with different default values can search the same request, including ID searches, without relying on JSON fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.