quickwit-oss / quickwit-oss/quickwit
Documents loose after their ingestion
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
After the ingestion 1.8T of data we lost 9 documents.
We log every response from quickwit during ingestion. And we see in logs
{"num_docs_for_processing":7000} #247545 times
{"num_docs_for_processing":1000} #2 times
{"num_docs_for_processing":71} #1 time
So we expected 1732817071 docs as a result but got 1732817062
# curl -H "Content-type: application/json" -X POST \
> http://localhost:7280/api/v1/taxi/search/ \
> -d '{"query":"*","max_hits":0,"aggs":{"count(*)":{"value_count":{"field":"id"}}}}'
{
"num_hits": 1732817062,
"hits": [],
"elapsed_time_micros": 1679406,
"errors": [],
"aggregations": {
"count(*)": {
"value": 1732817062.0
}
}
}
Steps to reproduce (if applicable)
This is a big amount of data so we can't provide the dump easily.
You can reproduce this issue via databases comparing tool
- Clone comparing tool
git clone git@github.com:db-benchmarks/db-benchmarks.git
cd db-benchmarks
git checkout feat/quickwit
- Copy
.env.exampleto.env - Update
cpusetin.envwith the default value of CPUs that your machine has - Open the test folder
cd tests/taxi
- Add
exit 1to prevent other engines init (It doesn't affect our issue and save us space) - Run
./init
Ingestion will take 3-4 days after you will see the problem.
Expected behavior
1732817071 count of docs as results
Configuration:
Please provide:
- Output of
quickwit --version0.8.1 - The index_config.yaml
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/taxi/quickwit/index-config.yaml and the db-benchmarks tests/taxi/init entry point; review the Quickwit 0.8.1 setup and run ./init using the documented initialization change. Done means reproducing the ingestion count discrepancy and narrowing the missing documents to a reproducible component or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100