quickwit-oss / quickwit-oss/quickwit
Force commit might not work with v2 ingest CLI command
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
Given that ingest V2 might send different requests to api/v1/<index id>/ingest/ingest-v2 to different shards, it isn't guarantied that the various requests will be picked up by the same indexer or in order. But the quickwit index ingest CLI command breaks down its input data stream into batches and only assigns the force and wait_for commit-type to the last batch.
Steps to reproduce (if applicable)
This is pretty hard to reproduce, it might happen when ingesting large files (batch size limit is 10MB) on a busy multi-indexer cluster (otherwise it is unlikely for shards to be processed out of order)
Expected behavior
After returning, all documents processed by quickwit index ingest should be indexed and committed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the quickwit index ingest CLI command and trace how batched requests assign commit-type across the ingest-v2 endpoint. Run it with large input against a busy multi-indexer setup if available, and consider the work done when the command returns only after all processed documents are indexed and committed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100