lance-format / lance-format/lance-ray
btree distributed build cause fragment not ordered in manifest
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 48
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 5
Description
User reported bug, creating this for tracking purpose:
On a smaller table I was able to create a btree index, but on a larger table I was running into an error. I was wondering if anyone had any ideas before I dug deeper. The error was assertion failed: the_id as i64 > last_id
Looks like somehow the fragments are not ordered in the manifest. Very likely that in create_index(fragment_ids=<...>) it's not sorted. We probably can look at how ray is doing the create_scalar_index, maybe it's somehow passing arbitrary order.
cc @chenghao-guo @xloya
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rust/lance/src/dataset.rs around the manifest handling referenced in the report, then trace how create_index(fragment_ids=...) is called by the Ray integration and compare it with Ray's create_scalar_index path. Reproduce the larger-table btree failure and confirm that the resulting manifest fragments are ordered so the assertion no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100