quickwit-oss / quickwit-oss/quickwit

make delete task hammer less the metastore

Open
#4,560 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

currently on startup, the delete task does quite a few request to the metastore. We could optimize a bit so less calls are made, and also make it so they are not made all at once. Here are some ideas:

  • DeleteTaskPipeline::spawn_pipeline could be called while holding a semaphore so there aren't 1k requests all at once.
  • get_relevant_stale_splits() could be made to accept None as delete opstamp, and figure itself what is the latest opstamp and return it along, saving one call to last_delete_opstamp() per index.
  • in partition_splits_by_deletes, call list_delete_tasks() only once, and sort ourselves which delete have already been applied to any given split

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing DeleteTaskPipeline::spawn_pipeline, get_relevant_stale_splits(), last_delete_opstamp(), partition_splits_by_deletes, and list_delete_tasks(). Compare the startup request pattern with the proposed semaphore and call-reduction changes; done means fewer metastore requests and no burst of all requests at once.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.