lance-format / lance-format/lance
Create a index lifecycle test
Open
Nobody has claimed this yet.
ci
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Create an integration test that validates it can handle a sequence of writes and optimization steps.
- Create a compaction rule (run compaction when there are > 5 fragments)
- Create a optimize_indices rule (create a new index segment when there are > 100 unindexed rows, merge index when there are > 3 segments)
- Have a test that:
- Creates a dataset with starting rows and index
- In a loop:
- Perform an upsert
- Apply compaction rule
- Apply optimize_indices rule
This would be to catch bugs like https://github.com/lance-format/lance/issues/5312. (Should validate it would have caught this.)
We should run this for all index types.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the related issues 5312 and 4034, then locate the existing integration tests and index-type test coverage. Implement the described write, compaction, and optimize_indices sequence for every index type, and verify that the test reproduces the failure discussed in issue 5312.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100