quickwit-oss / quickwit-oss/quickwit
[Discussion] Keeping or removing the check on index existence in `list_splits` method
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Currently, in list_splits, we return filtered splits notably with a filter on index UIDs.
In cases where an index does not exist, we have a different behavior between the file-backed metastore and the PostgreSQL metastore.
- file-backed metastore: as soon as an index is not found, it returns an error
IndexesDoNotExist. - PostgreSQL metastore: we check only indexes' existence if no splits are returned. See the convoluted code.
(note that the PR on the multi-indexes #3734 introduced this discrepancy).
Also, with the multi indexes features, querying splits while a user deletes some indexes will more likely happen (scenario such as "a user makes a search query on a list of indexes, an admin user deletes one of the indexes, the user gets an error").
One pro for returning an error when an index does not exist is that the caller gets the right error immediately. This is useful in Quickwit's code base as we can react to this error and stop getting splits from the metastore for a given index.
This is why I'm currently reluctant to drop the existence check.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the list_splits method and the PostgreSQL implementation in quickwit/quickwit-metastore/src/metastore/postgresql_metastore.rs, especially lines 762–786, then compare its index-existence handling with the file-backed metastore. Resolve whether both metstores should return the same result for missing indexes, and consider the multi-index deletion scenario before documenting the decision and aligning the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100