quickwit-oss / quickwit-oss/quickwit
Target optimal num docs by split when ingesting data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
We can ingest data from a file with the CLI. Currently, it produces splits generally with num docs << 10M which is our general optimal num docs. This comes from the fact that we commit a split every 60 seconds (default commit_timeout_secs). This will lead to merges and will lower the indexing speed.
Ideally, we would need to produce split with num docs of 10M directly, this can be done by putting a high commit_timeout_secs by default.
I suggest putting the commit timeout to 3600 seconds to avoid merges.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the CLI data-ingestion entry point and locate the default commit_timeout_secs setting. Check how the timeout controls split commits, change the default to 3600 seconds, and verify that file ingestion uses the new default without requiring an explicit setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100