quickwit-oss / quickwit-oss/quickwit

Improve dev XP on index create/update

Open
#2,370 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Currently, the dev XP is quite poor when a user needs to create an index or make some updates on one index.

I played a bit with Quickwit on a Kubernetes cluster and I'm just putting here my frustrations

Create an index

To create an index, I need to connect to the metastore pod which typically has the right env env variables to execute correctly a CLI command against the right database. Then I need to cp my local index config and source files on the pod and execute the right CLI commands.

Then I need to restart my indexers.

Update an index

All these modifications require to use of SQL queries to modify the index_metadata_json. Then I need to restart the indexers.

Possible solutions

For updates, we could provide a CLI command to update the index config (with some checks on some restrictions to avoid breaking everything). This solution has one drawback: we need to connect to the metastore pod which typically has the right env env variables to execute correctly a CLI command against the right database.

Another solution that could help on create/update could be to provide a REST endpoint and let the user restart the indexers if needed.

I have a preference for providing a REST endpoint... we always need to restart the indexers anyway :/ (we could poll the metastore on the indexer side to check if there are new indexes).
I know we will handle things correctly with the control place but for now, I think it is not so crazy to add the right endpoint.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by mapping the current Kubernetes metastore workflow and the SQL updates to index_metadata_json, then compare the proposed CLI and REST approaches. Done would require an agreed scope and an implemented create/update path with appropriate checks and indexer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust, sql
Domain
backend-api-design, devops, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.