quickwit-oss / quickwit-oss/quickwit
delete operation can delete documents not yet ingested
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
in some cases, creating a deletion task can delete document which have not yet been ingested. This happen because when merging splits, the smallest delete_opstamp of any split in the merge is the delete_opstamp of the resulting merged split
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
- ingest a document, its split get a
delete_opstampof 0 - send a delete operation which would delete that document
- ingest the same document again, its split get a
delete_opstampof 1 - search: both document appear
- wait a bit for a merge
- verify list of splits: both splits got merged, the resulting split has a
delete_opstampof 0 - wait for
maturation_period(possibly, lower it before creating the index) - both documents were deleted.
Expected behavior
The delete operation shouldn't delete documents not yet ingested. However it should also make sure it (eventually) deletes documents previously ingested, so setting new_delete_opstamp = max(delete_opstamps) (instead of min) is not a solution. A proper solution would probably require to execute the delete operation on splits just before/while merging them, so the delete_opstamp of the merged split represent the "true" state of the split
Workaround:
Delete based only on something that could qualify as a unique id, or add a clause in the delete operation to limit its effect only to document with a timestamp lower than now(), assuming documents are ingested roughly in order and have such a field.
Configuration:
Please provide:
- Output of
quickwit --version:Quickwit 0.6.1-nightly (8f314f5 2023-07-04T13:52:10Z) - The index_config.yaml
version: 0.4
index_id: delete-test
doc_mapping:
field_mappings:
- name: id
type: text
fast: true
- name: type
type: text
fast: true
tokenizer: raw
- name: public
type: bool
fast: true
- name: payload
type: json
record: position
tokenizer: default
- name: body
type: text
record: position
tokenizer: default
- name: org
type: json
tokenizer: default
- name: repo
type: json
tokenizer: default
- name: actor
type: json
tokenizer: default
- name: other
type: json
tokenizer: default
- name: created_at
type: datetime
fast: true
input_formats:
- rfc3339
precision: seconds
timestamp_field: created_at
indexing_settings:
commit_timeout_secs: 2
split_num_docs_target: 5000
merge_policy:
type: "stable_log"
merge_factor: 2
max_merge_factor: 4
maturation_period: 1 hours
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the supplied index_config.yaml and the seven-step sequence, then inspect the split merge and delete-operation handling around delete_opstamp and maturation_period. Done means a delete operation removes previously ingested matching documents without deleting documents ingested afterward, including after the affected splits are merged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100