quickwit-oss / quickwit-oss/quickwit
Improve dev XP on index create/update
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Currently, the dev XP is quite poor when a user needs to create an index or make some updates on one index.
I played a bit with Quickwit on a Kubernetes cluster and I'm just putting here my frustrations
Create an index
To create an index, I need to connect to the metastore pod which typically has the right env env variables to execute correctly a CLI command against the right database. Then I need to cp my local index config and source files on the pod and execute the right CLI commands.
Then I need to restart my indexers.
Update an index
- update the
max_merge_write_throughput - update
max_num_partitions - update the merge policy
- update the
commit_timeout_secs - update retention policy (https://github.com/quickwit-oss/quickwit/pull/4832)
All these modifications require to use of SQL queries to modify the index_metadata_json. Then I need to restart the indexers.
Possible solutions
For updates, we could provide a CLI command to update the index config (with some checks on some restrictions to avoid breaking everything). This solution has one drawback: we need to connect to the metastore pod which typically has the right env env variables to execute correctly a CLI command against the right database.
Another solution that could help on create/update could be to provide a REST endpoint and let the user restart the indexers if needed.
I have a preference for providing a REST endpoint... we always need to restart the indexers anyway :/ (we could poll the metastore on the indexer side to check if there are new indexes).
I know we will handle things correctly with the control place but for now, I think it is not so crazy to add the right endpoint.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by mapping the current Kubernetes metastore workflow and the SQL updates to index_metadata_json, then compare the proposed CLI and REST approaches. Done would require an agreed scope and an implemented create/update path with appropriate checks and indexer behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust, sql
- Domain
- backend-api-design, devops, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100