quickwit-oss / quickwit-oss/quickwit

Mutability / Overwriting resources / `id` field / ingest resources using delete task

Open
#3,081 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Is your feature request related to a problem? Please describe.
I understand that QuickWit achieves some of its awesome benefits (scale, performance) by not having to deal with updating resources.

This makes QuickWit not a good option if your data changes a lot. But what if your data changes only sometimes? Then you'd have to implement your own replace event which itself creates a delete task for some resource and after that ingests the data again.

I can totally understand if this is not a problem you'd want to address, as it does not seem like a use case for the logs market. However, if you have ambitions to make the software more powerful outside of the logs domain, It would be a nice addition.

Describe the solution you'd like
Ideally, QuickWit would use a pre-defined id field to check if the resource needs to be overwritten instead of appended. However, I'm assuming sure that is not technically feasible and would slow things down too much. So here's a simpler option:

Ingest field in delete tasks

So here's a perhaps simpler alternative: allow an optional ingest field to the delete task JSON, which ingests the data after succesfully having performed the deletion query. This effectively is a replace command.

I suppose implementing this means the DeleteQuery and DeleteTask structs should get an Option<DocBatch> field (or maybe Vec<DocBatch>).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the DeleteQuery and DeleteTask structs and the delete-task JSON processing path; the issue suggests adding an optional DocBatch or Vec field. Check how deletion completes before determining where follow-up ingestion belongs. Done means a delete task can optionally ingest its supplied data after a successful deletion, while existing delete tasks remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.