quickwit-oss / quickwit-oss/quickwit
Ingest_v2 MergePipeline (Packager, Uploader) inconsistency after index update
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
-
Packager
If you use the tags/partition feature, after a doc_mapping update that adds new fields and shifts the initial positions of fields responsible for tags/partitions, the split metadata will become corrupted (as described in #5866). This happens because the savedNamedFielddoes not respect the schema of the split. For the old MergePipeline, this means that all new splits will not be found via tag-based search. For a recreated pipeline, it could drop existing tags if the new position of the tag field points to a field in the old index that contains> MAX_VALUES_PER_TAG_FIELDterms. -
Uploader
After #5543, the Uploader depends on the RetentionPolicy. However, updating the policy does not update the pipelines. For the IndexingPipeline, this can be done easily by patchingindexing_params_fingerprint(), but it does not affect the MergePipeline since the old one may still be used. This is not critical, but leads to the same noisy errors as in the original issue if you setnew_retention < maturation_period < old_retention.
P.S. It seems there is a redundant lock on the indexes table in the Postgres metastore's publish_splits() with Ingest_v2, since index_metadata is not updated.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Ingest_v2 MergePipeline's Packager and Uploader, then inspect IndexingPipeline and indexing_params_fingerprint() for the retention-policy behavior described. Review the Postgres metastore's publish_splits() as a separate lead. Done means index updates preserve split metadata and retention changes no longer produce the reported noisy errors, with the lock behavior clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, rust
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100