quickwit-oss / quickwit-oss/quickwit
Control plane restart
Open
Nobody has claimed this yet.
bug
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Observed on cicada : On control plane restart, upon reception of the chitchat state, we end up removing a lot of shards.
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z INFO quickwit_control_plane::model: removing shards from model source_uid=app-logs-649:01HP6FTGHCS28TTAN2SMJPP9FV:_ingest-source shard_ids=[ShardId("01HVRBQ92NRTPBHCV4V7VKBK3Z")]
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z INFO quickwit_control_plane::control_plane: received eof shard via gossip shard_id=01HVRKG1BCBBKV5KVWWQYC1P52 position=Position::Eof
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z INFO quickwit_control_plane::control_plane: deleting shards shard_ids=[ShardId("01HVRKG1BCBBKV5KVWWQYC1P52")] source_uid=SourceUid { index_uid: IndexUid { index_id: "app-logs-346", incarnation_id: Ulid(2064201455267077262770700750118270741) }, source_id: "_ingest-source" }
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z INFO quickwit_control_plane::model: removing shards from model source_uid=app-logs-346:01HP6FTGE8NG862NPA8J06ADRN:_ingest-source shard_ids=[ShardId("01HVRKG1BCBBKV5KVWWQYC1P52")]
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z INFO quickwit_control_plane::control_plane: received eof shard via gossip shard_id=01HVRKG1BCBF76A11B833W3SX8 position=Position::Eof
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z INFO quickwit_control_plane::control_plane: deleting shards shard_ids=[ShardId("01HVRKG1BCBF76A11B833W3SX8")] source_uid=SourceUid { index_uid: IndexUid { index_id: "app-logs-346", incarnation_id: Ulid(2064201455267077262770700750118270741) },
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the quickwit_control_plane::control_plane gossip handling and quickwit_control_plane::model shard-removal paths shown in the logs. Reproduce a control-plane restart with the chitchat state and compare the resulting shard state; done means valid shards are no longer removed during recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100