lance-format / lance-format/lance
Follow-up: remove per-batch offsets copy in variable full-zip decode
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
PR #6013 fixes a scan-full regression by slicing variable data per batch and rebasing offsets to 0.
As noted in review, downstream variable decompressors should theoretically be able to use offsets to determine the required decode range, which could let us avoid the per-batch offsets copy introduced in the fix.
The current fix is correct and improves performance, but we should investigate whether we can remove this extra offsets copy safely.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #6013 and its scan-full regression fix, focusing on the variable full-zip decode path and the per-batch offsets copy. Determine whether downstream variable decompressors can safely use offsets for the required decode range, while preserving the current fix's correctness and performance improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100