quickwit-oss / quickwit-oss/quickwit
Ingestion bug when using partition key and unsorted data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
Cant ingest data while using a partition key if number of partition key is high and ndjson data is not sorted per partition key.
Steps to reproduce (if applicable)
Ingest 100M more docs with partition key and unsorted data with default configuration
Expected behavior
Data to be ingested in the index without any issue.
Configuration:
Default configuration, version 0.8.1, please check this thread (https://discord.com/channels/908281611840282624/1233364603480576092) where we discuss the issue and hopefully find a solution for further details
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is identified in the issue. Start by reproducing ingestion of over 100M unsorted NDJSON documents with a high-cardinality partition key under the default configuration, then review the linked discussion for additional context; done means the data ingests into the index without failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100