quickwit-oss / quickwit-oss/quickwit

Persistent "no open shard found on ingester" after some indexer restart

Open
#6,531 13 comments 3 reactions 1 assignee View on GitHub

@dayaffe is already working on this.

Since Sep 9, 2026.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug

After some indexer restart (for example auto-scaling/k8s repacking events), I'm sometimes getting a persistent stream of no open shard found on ingester errors. A rolling restart of all the indexers sometimes appears to fix it.

Steps to reproduce (if applicable)

Observed on a production environment (the restart happened at 7:03 CET/5:03 UTC):

Image

Attached selected logs at the time of the restart (filtering out all the ingestion traffic, focusing on cluster events):

extract-2026-06-19T08_26_57.708Z.csv

Expected behavior

The no open shard found on ingester errors don't persist forever.

Configuration:

  1. quickwit version: 0.9.0 (x86_64-unknown-linux-gnu 2026-04-19T08:54:33Z e1732a7) (the latest edge from month ago)
  2. a few indexers, a single index taking 99% of the traffic, min_shards: 2.
  3. constant traffic (i.e. the shard almost never idle and closed by CloseIdleShardsTask)
  4. almost no calls to get_or_create_open_shards (1 call/ hour)

Tentative analysis:

I'm not 100% sure, but the logs would seem to point at some stalled routing tables in the other indexers

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.