quickwit-oss / quickwit-oss/quickwit
Improve pulsar source stability when using partitioned topics
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Currently, Pulsar has a default failover timeout of 1s, when combined with partitioned topics in order to allow for distributed indexing, it is possible for indexers to become unstable and unable to make progress in the event that there is high network latency or an external factor causing the failover to trigger.
This is because we rely on the checkpoints to ensure the exactly-once semantics, which currently will error and restart the whole pipeline when topic partitions are assigned to indexers (this happens whenever an indexer connects/disconnects), when this restart occurs if the pipeline does not restart within 1s or so, Pulsar will re-assign the partitions and replay unacknowledged (documents not yet published by quickwit) causing a cycle to begin of partitions being assigned, re-assigned, erroring, restarting and repeating.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Pulsar source's checkpoint handling and pipeline restart path described in the issue. Reproduce or inspect behavior with partitioned topics and the 1s failover timeout; done means partition reassignment no longer causes an error/restart loop while preserving exactly-once semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100