quickwit-oss / quickwit-oss/quickwit
Duplicate document prevention by specifying id
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
In elasticsearch we can specify which field is the id.
This prevents duplicate of documents when re-initializing log ingestion which would otherwise create duplicate docs.
Is this planned in quickwit without having the client calling quickwit to get all the unique ids and send only unique ids not present in logs?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. First clarify which ingestion paths and identifier semantics should be supported, then inspect Quickwit’s ingestion and indexing entry points for existing document-ID handling. Done should mean reinitializing log ingestion does not create duplicate documents without client-side lookup of existing IDs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100