quickwit-oss / quickwit-oss/quickwit

Ingest_v2 MergePipeline (Packager, Uploader) inconsistency after index update

Open
#6,155 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

  1. Packager
    If you use the tags/partition feature, after a doc_mapping update that adds new fields and shifts the initial positions of fields responsible for tags/partitions, the split metadata will become corrupted (as described in #5866). This happens because the saved NamedField does not respect the schema of the split. For the old MergePipeline, this means that all new splits will not be found via tag-based search. For a recreated pipeline, it could drop existing tags if the new position of the tag field points to a field in the old index that contains > MAX_VALUES_PER_TAG_FIELD terms.

  2. Uploader
    After #5543, the Uploader depends on the RetentionPolicy. However, updating the policy does not update the pipelines. For the IndexingPipeline, this can be done easily by patching indexing_params_fingerprint(), but it does not affect the MergePipeline since the old one may still be used. This is not critical, but leads to the same noisy errors as in the original issue if you set new_retention < maturation_period < old_retention.

P.S. It seems there is a redundant lock on the indexes table in the Postgres metastore's publish_splits() with Ingest_v2, since index_metadata is not updated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Ingest_v2 MergePipeline's Packager and Uploader, then inspect IndexingPipeline and indexing_params_fingerprint() for the retention-policy behavior described. Review the Postgres metastore's publish_splits() as a separate lead. Done means index updates preserve split metadata and retention changes no longer produce the reported noisy errors, with the lock behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, rust
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.