quickwit-oss / quickwit-oss/quickwit

Improve pulsar source stability when using partitioned topics

Open
#2,795 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Currently, Pulsar has a default failover timeout of 1s, when combined with partitioned topics in order to allow for distributed indexing, it is possible for indexers to become unstable and unable to make progress in the event that there is high network latency or an external factor causing the failover to trigger.

This is because we rely on the checkpoints to ensure the exactly-once semantics, which currently will error and restart the whole pipeline when topic partitions are assigned to indexers (this happens whenever an indexer connects/disconnects), when this restart occurs if the pipeline does not restart within 1s or so, Pulsar will re-assign the partitions and replay unacknowledged (documents not yet published by quickwit) causing a cycle to begin of partitions being assigned, re-assigned, erroring, restarting and repeating.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Pulsar source's checkpoint handling and pipeline restart path described in the issue. Reproduce or inspect behavior with partitioned topics and the 1s failover timeout; done means partition reassignment no longer causes an error/restart loop while preserving exactly-once semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.