quickwit-oss / quickwit-oss/quickwit

delete operation can delete documents not yet ingested

Open
#3,612 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
in some cases, creating a deletion task can delete document which have not yet been ingested. This happen because when merging splits, the smallest delete_opstamp of any split in the merge is the delete_opstamp of the resulting merged split

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. ingest a document, its split get a delete_opstamp of 0
  2. send a delete operation which would delete that document
  3. ingest the same document again, its split get a delete_opstamp of 1
  4. search: both document appear
  5. wait a bit for a merge
  6. verify list of splits: both splits got merged, the resulting split has a delete_opstamp of 0
  7. wait for maturation_period (possibly, lower it before creating the index)
  8. both documents were deleted.

Expected behavior
The delete operation shouldn't delete documents not yet ingested. However it should also make sure it (eventually) deletes documents previously ingested, so setting new_delete_opstamp = max(delete_opstamps) (instead of min) is not a solution. A proper solution would probably require to execute the delete operation on splits just before/while merging them, so the delete_opstamp of the merged split represent the "true" state of the split

Workaround:
Delete based only on something that could qualify as a unique id, or add a clause in the delete operation to limit its effect only to document with a timestamp lower than now(), assuming documents are ingested roughly in order and have such a field.

Configuration:
Please provide:

  1. Output of quickwit --version: Quickwit 0.6.1-nightly (8f314f5 2023-07-04T13:52:10Z)
  2. The index_config.yaml
version: 0.4

index_id: delete-test

doc_mapping:
  field_mappings:
    - name: id
      type: text
      fast: true
    - name: type
      type: text
      fast: true
      tokenizer: raw
    - name: public
      type: bool
      fast: true
    - name: payload
      type: json
      record: position
      tokenizer: default
    - name: body
      type: text
      record: position
      tokenizer: default
    - name: org
      type: json
      tokenizer: default
    - name: repo
      type: json
      tokenizer: default
    - name: actor
      type: json
      tokenizer: default
    - name: other
      type: json
      tokenizer: default
    - name: created_at
      type: datetime
      fast: true
      input_formats:
        - rfc3339
      precision: seconds
  timestamp_field: created_at

indexing_settings:
  commit_timeout_secs: 2
  split_num_docs_target: 5000
  merge_policy:
    type: "stable_log"
    merge_factor: 2
    max_merge_factor: 4
    maturation_period: 1 hours

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the supplied index_config.yaml and the seven-step sequence, then inspect the split merge and delete-operation handling around delete_opstamp and maturation_period. Done means a delete operation removes previously ingested matching documents without deleting documents ingested afterward, including after the affected splits are merged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.