quickwit-oss / quickwit-oss/quickwit

Deleting and recreating a topic for an index result in retries on a non-recoverable scenario

Open
#1,602 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
If you delete and recreate the topic backing a Kafka source after reading data through it the server goes into a non-recoverable retry loop due to missing offsets.

2022-06-02T21:21:54.244Z ERROR {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=2}::{msg_id=148}: quickwit_indexing::actors::indexing_pipeline: Error while spawning indexing pipeline, retrying after some time. error=Failed to create source `quickwit-kafka-test` of type `kafka`. Cause: Last checkpointed offset `99999` is greater or equal to high watermark `0`.

Caused by:
    Last checkpointed offset `99999` is greater or equal to high watermark `0`. retry_count=5 retry_delay=64s

If you then use this topic and push more data through it I would expect this might start working again after getting to 100,000 records but it will result in the loss of the first 100,000 records since offsets will restart from 0.

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. setup a kafka source
  2. push some data through it so you have committed offsets
  3. delete the topic
  4. recreate the topic
  5. restart the server and you get a continually retrying error

Expected behavior
This isn't really a recoverable scenario without using offset reset logic like the Kafka high level consumer uses. In this case the correct scenario is likely to detect the missing offset and reset to the low watermark for the topic.

You're also going to have a similar problem if a topic has retention configured and the data expires before Quickwit reads it. The low watermark on the topic in that case will be higher than the committed offset and the offset will need to be moved forward.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Kafka source's checkpoint and offset handling, using the deletion and recreation steps to reproduce the retry loop. Compare the committed offset with the topic's low and high watermarks; done means missing or expired offsets advance or reset instead of causing endless retries, without silently losing unread records.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.