quickwit-oss / quickwit-oss/quickwit
Stream splits in file backed metastore `list_splits`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
In the file-backed metastore implementation, list_splits first builds a Vec<Split> by calling the inner_list_splits function and then builds a stream of split chunks out of it.
Instead, we could avoid building the Vec<Split>, update inner_list_splits to return a stream of splits, and finally make list_splits return a stream of groups of splits.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in quickwit/quickwit-metastore/src/metastore/file_backed_metastore/mod.rs at list_splits and inner_list_splits, using the linked lines as the entry point. Trace the existing stream chunking flow and verify that the completed refactor avoids the intermediate Vec while preserving the grouped stream returned by list_splits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100