quickwit-oss / quickwit-oss/quickwit

Documents loose after their ingestion

Open
#5,465 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
After the ingestion 1.8T of data we lost 9 documents.
We log every response from quickwit during ingestion. And we see in logs

{"num_docs_for_processing":7000} #247545 times
{"num_docs_for_processing":1000} #2 times
{"num_docs_for_processing":71} #1 time

So we expected 1732817071 docs as a result but got 1732817062

# curl -H "Content-type: application/json" -X POST \
> http://localhost:7280/api/v1/taxi/search/ \
> -d '{"query":"*","max_hits":0,"aggs":{"count(*)":{"value_count":{"field":"id"}}}}'
{ 
  "num_hits": 1732817062,
  "hits": [],
  "elapsed_time_micros": 1679406,
  "errors": [],
  "aggregations": {
    "count(*)": {
      "value": 1732817062.0
    }
  }
}

Steps to reproduce (if applicable)

This is a big amount of data so we can't provide the dump easily.

You can reproduce this issue via databases comparing tool

  1. Clone comparing tool
git clone git@github.com:db-benchmarks/db-benchmarks.git
cd db-benchmarks
git checkout feat/quickwit
  1. Copy .env.example to .env
  2. Update cpuset in .env with the default value of CPUs that your machine has
  3. Open the test folder
cd tests/taxi
  1. Add exit 1 to prevent other engines init (It doesn't affect our issue and save us space)
  2. Run ./init

Ingestion will take 3-4 days after you will see the problem.

Expected behavior
1732817071 count of docs as results

Configuration:
Please provide:

  1. Output of quickwit --version 0.8.1
  2. The index_config.yaml

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/taxi/quickwit/index-config.yaml and the db-benchmarks tests/taxi/init entry point; review the Quickwit 0.8.1 setup and run ./init using the documented initialization change. Done means reproducing the ingestion count discrepancy and narrowing the missing documents to a reproducible component or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.