quickwit-oss / quickwit-oss/quickwit

Add logs on delete index / create with overwrite option to show quickwit is working during index deletion

Open
#2,151 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Deleting an index with thousands of splits can take a long time (several minutes). You can believe that quickwit is doing nothing, which can be frustrating for a user.

We should a log like "deleting XXXXX splits, this can takes minutes... we can even add a progress bar :). Also, on prompt, we could also warn the user that the operation can last several minutes if there are tens of thousands of splits.

Just adding the current logs on creation with overwrite flag for context:

quickwit-latest index create --index-config ./config/index_config.yaml --config config/indexer_config.yaml --overwrite
2022-10-20T07:31:17.297Z  INFO quickwit: version="0.3.1-nightly" commit="d6b1d30be"
2022-10-20T07:31:17.299Z  INFO quickwit_config::config: Using sniffed advertise address. advertise_address=10.65.50.61 interface_name=ens5
2022-10-20T07:31:17.299Z  INFO quickwit_config::config: Using sniffed advertise address. advertise_address=10.65.50.61 interface_name=ens5
2022-10-20T07:31:17.300Z  INFO quickwit_cli: Loaded Quickwit config. config_uri=file:///home/fmassot/config/indexer_config.yaml config=QuickwitConfig { version: 0, cluster_id: "quickwit-cluster", node_id: "indexer-0", rest_listen_addr: 0.0.0.0:8080, gossip_listen_addr: 0.0.0.0:8080, grpc_listen_addr: 0.0.0.0:8081, gossip_advertise_addr: 10.65.50.61:8080, grpc_advertise_addr: 10.65.50.61:8081, enabled_services: {Indexer, Metastore}, peer_seeds: ["ip-10-65-49-64.eu-west-1.compute.internal:8080"], metastore_uri: Uri { uri: "postgres://quickwit:***redacted***@localhost/poc_quickwit" }, default_index_root_uri: Uri { uri: "s3://cs-external-providers-resources-eu-west-1/quickwit/" }, data_dir_path: "/ephemeral2/quickwit-indexing", indexer_config: IndexerConfig { enable_opentelemetry_otlp_service: false, split_store_max_num_bytes: Byte(150000000000), split_store_max_num_splits: 10000, max_concurrent_split_uploads: 100 }, searcher_config: SearcherConfig { enable_jaeger_service: false, fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_searches: 100, max_num_concurrent_split_streams: 100 } }
2022-10-20T07:31:17.398Z  INFO sniff_aws_region: quickwit_aws::region: set-aws-region region=EuWest1 from="ec2-instance-metadata-service"
✔ This operation will overwrite the index `quickwit-fts-staging` and delete all its data. Do you want to proceed? · yes

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the CLI paths for index deletion and the create --overwrite confirmation, then trace where long-running split deletion is invoked. Review the existing creation logs and prompt behavior. Done means users receive useful progress or duration guidance while deletion is running and a warning about lengthy overwrite operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.