lance-format / lance-format/lance
Add a min_batch_size property to the scanner
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The current batch size is a max_batch_size only. It will slice large batches but will not merge smaller batches. There are cases where it is important for users to have exact sized batches (e.g. some training software will fill in small batches with dummy data / padding and users want to avoid that). We can be clear in the documentation that setting the property could have a performance impact.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the scanner implementation and its existing max_batch_size handling, then read how batches are split and exposed to users. Done means a min_batch_size property can merge smaller batches to the requested size, with documentation warning about possible performance impact and coverage for the new behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100