quickwit-oss / quickwit-oss/quickwit
consider not retuning all columns from metastore when listing splits
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
splits has 12 columns, while we filter based on most of these, we rarely need to fetch more than split_id index_uid and possibly split_metadata_json. We might reduce some postgres + metastore load by not fetching other columns, especially tags which can grow fairly large and often uses toast when it's not empty
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the metastore path used when listing splits and identify how the splits columns are selected and which columns the filters require. Confirm that listing still supports filtering while fetching only split_id, index_uid, and split_metadata_json when needed, and measure or test that unnecessary columns such as tags are no longer loaded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100