quickwit-oss / quickwit-oss/quickwit

[Discussion] Keeping or removing the check on index existence in `list_splits` method

Open
#3,760 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Currently, in list_splits, we return filtered splits notably with a filter on index UIDs.

In cases where an index does not exist, we have a different behavior between the file-backed metastore and the PostgreSQL metastore.

  • file-backed metastore: as soon as an index is not found, it returns an error IndexesDoNotExist.
  • PostgreSQL metastore: we check only indexes' existence if no splits are returned. See the convoluted code.

(note that the PR on the multi-indexes #3734 introduced this discrepancy).

Also, with the multi indexes features, querying splits while a user deletes some indexes will more likely happen (scenario such as "a user makes a search query on a list of indexes, an admin user deletes one of the indexes, the user gets an error").

One pro for returning an error when an index does not exist is that the caller gets the right error immediately. This is useful in Quickwit's code base as we can react to this error and stop getting splits from the metastore for a given index.
This is why I'm currently reluctant to drop the existence check.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the list_splits method and the PostgreSQL implementation in quickwit/quickwit-metastore/src/metastore/postgresql_metastore.rs, especially lines 762–786, then compare its index-existence handling with the file-backed metastore. Resolve whether both metstores should return the same result for missing indexes, and consider the multi-index deletion scenario before documenting the decision and aligning the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.