quickwit-oss / quickwit-oss/quickwit
Deleting and recreating a topic for an index result in retries on a non-recoverable scenario
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
If you delete and recreate the topic backing a Kafka source after reading data through it the server goes into a non-recoverable retry loop due to missing offsets.
2022-06-02T21:21:54.244Z ERROR {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=2}::{msg_id=148}: quickwit_indexing::actors::indexing_pipeline: Error while spawning indexing pipeline, retrying after some time. error=Failed to create source `quickwit-kafka-test` of type `kafka`. Cause: Last checkpointed offset `99999` is greater or equal to high watermark `0`.
Caused by:
Last checkpointed offset `99999` is greater or equal to high watermark `0`. retry_count=5 retry_delay=64s
If you then use this topic and push more data through it I would expect this might start working again after getting to 100,000 records but it will result in the loss of the first 100,000 records since offsets will restart from 0.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
- setup a kafka source
- push some data through it so you have committed offsets
- delete the topic
- recreate the topic
- restart the server and you get a continually retrying error
Expected behavior
This isn't really a recoverable scenario without using offset reset logic like the Kafka high level consumer uses. In this case the correct scenario is likely to detect the missing offset and reset to the low watermark for the topic.
You're also going to have a similar problem if a topic has retention configured and the data expires before Quickwit reads it. The low watermark on the topic in that case will be higher than the committed offset and the offset will need to be moved forward.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Kafka source's checkpoint and offset handling, using the deletion and recreation steps to reproduce the retry loop. Compare the committed offset with the topic's low and high watermarks; done means missing or expired offsets advance or reset instead of causing endless retries, without silently losing unread records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100