quickwit-oss / quickwit-oss/quickwit
Mutability / Overwriting resources / `id` field / ingest resources using delete task
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Is your feature request related to a problem? Please describe.
I understand that QuickWit achieves some of its awesome benefits (scale, performance) by not having to deal with updating resources.
This makes QuickWit not a good option if your data changes a lot. But what if your data changes only sometimes? Then you'd have to implement your own replace event which itself creates a delete task for some resource and after that ingests the data again.
I can totally understand if this is not a problem you'd want to address, as it does not seem like a use case for the logs market. However, if you have ambitions to make the software more powerful outside of the logs domain, It would be a nice addition.
Describe the solution you'd like
Ideally, QuickWit would use a pre-defined id field to check if the resource needs to be overwritten instead of appended. However, I'm assuming sure that is not technically feasible and would slow things down too much. So here's a simpler option:
Ingest field in delete tasks
So here's a perhaps simpler alternative: allow an optional ingest field to the delete task JSON, which ingests the data after succesfully having performed the deletion query. This effectively is a replace command.
I suppose implementing this means the DeleteQuery and DeleteTask structs should get an Option<DocBatch> field (or maybe Vec<DocBatch>).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the DeleteQuery and DeleteTask structs and the delete-task JSON processing path; the issue suggests adding an optional DocBatch or Vec field. Check how deletion completes before determining where follow-up ingestion belongs. Done means a delete task can optionally ingest its supplied data after a successful deletion, while existing delete tasks remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100