lance-format / lance-format/lance
The StrictSizeBatchStream could be more efficient
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
From a review of an unrelated change:
Performance-wise, this seems a little unfortunate. It means that if we have batch_size=100 and a stream of 100 single row batches, we will call concat_batches 99 times to produce a batch. Would be nice to collect points to the batches and call concat_batches() one per output batch.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating StrictSizeBatchStream and its use of concat_batches. Trace how batches are accumulated for the configured batch_size, then verify that each output batch invokes concat_batches only once while preserving the existing stream results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100