quickwit-oss / quickwit-oss/quickwit
Sync and clean up local shards when an ingester starts
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
We have a one-way sync from the control plane to the ingesters, but the opposite path does not exist. In the case of Airmal, they resized their disks, which scratched the WALs. When the ingesters were restarted, they had no way to tell the CP that the shards they hosted no longer existed. The decommissioning logic should help prevent this scenario in the case of a graceful shutdown, but if, for some reason, it is not triggered, we can only fix the cluster state manually (clean up shards in metastore and restart CP).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ingester startup, the control-plane synchronization path, and the existing graceful-shutdown decommissioning logic. Inspect how shard state is stored in the metastore; done means a restarted ingester can report missing local shards so stale cluster state is cleaned up without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100