quickwit-oss / quickwit-oss/quickwit

splits xxxxxxx not found, failed to publish splits.

Open
#4,817 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
A clear and concise description of what the bug is.
I'm observing the error in the logs

quickwit-indexer-2 quickwit     splits `01HT2BDEQ76A92T8X010XD38G3` not found exit_status=Failure(failed to publish splits
quickwit-indexer-2 quickwit
quickwit-indexer-2 quickwit Caused by:
quickwit-indexer-2 quickwit     splits `01HT2BDEQ76A92T8X010XD38G3` not found)
quickwit-indexer-2 quickwit 2024-03-28T10:59:12.176Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Publisher-quiet-2Dom exit_status=failure(cause=failed to publish splits
quickwit-indexer-2 quickwit
quickwit-indexer-2 quickwit Caused by:
quickwit-indexer-2 quickwit     splits `01HT2BDEQ76A92T8X010XD38G3` not found)
quickwit-indexer-2 quickwit 2024-03-28T10:59:12.176Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=Publisher-quiet-2Dom exit_status=Failure(failed to publish splits
quickwit-indexer-2 quickwit
quickwit-indexer-2 quickwit Caused by:
quickwit-indexer-2 quickwit     splits `01HT2BDEQ76A92T8X010XD38G3` not found)

It resolves after sometime and then happen again. I tried to restart cluster but still no luck.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
It started to happen after upgrade to 0.8.0.

Expected behavior
No error message about splits don't exist.

I'm using gcs as storage and metastore as well.

  1. Output of quickwit --version
    Quickwit 0.8.0 (x86_64-unknown-linux-gnu 2024-03-18T15:20:23Z 7af4596)
  2. The index_config.yaml
{
  "commit_timeout_secs": 30,
  "docstore_compression_level": 8,
  "docstore_blocksize": 1000000,
  "split_num_docs_target": 10000000,
  "merge_policy": {
    "type": "stable_log",
    "min_level_num_docs": 100000,
    "merge_factor": 10,
    "max_merge_factor": 12,
    "maturation_period": "2days"
  },
  "resources": {
    "heap_size": "2.0 GB"
  }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the Publisher-quiet-2Dom actor error and the reported Quickwit 0.8.0 logs, then trace split publication when GCS is used for storage and the metastore. Reproduce the missing-split failure if possible and determine what behavior should prevent or recover from the publication error.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.