lance-format / lance-format/lance
bug: arrow offset overflow
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
I'm seeing some compactions fail with LanceError(Arrow): Offset overflow error: N, where N is always just over i32::MAX (observed ~ 2,150M). I've seen it come from 2 sites, both in the decode path where arrow_select::concat::concat stitches decoded pages into a single Arrow array: CompositeDecodeTask::decode at lance-encoding/src/array_encoding/logical/struct.rs:594 and StructuralCompositeDecodeArrayTask::decode at lance-encoding/src/encodings/logical/primitive.rs:4501 (and :4367). When one compaction's input carries more than 2 GiB of variable-width data in a single i32-offset column (Utf8/Binary/List), the concat overflows and the task fails. I've seen this on both lance 11.0.0-beta.22 and 12.0.0-beta.16;
Steps to reproduce
I can provide sample data that reproduces this, but it's too big to share here immediately.
Expected behavior
No response
Lance version
11.0.0-beta.22
Language binding
Rust
Environment
No response
Logs / traceback
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lance-encoding/src/array_encoding/logical/struct.rs at CompositeDecodeTask::decode:594 and lance-encoding/src/encodings/logical/primitive.rs at StructuralCompositeDecodeArrayTask::decode:4367 and :4501. Reproduce with variable-width Utf8, Binary, or List data exceeding 2 GiB in one i32-offset column, using the reporter's sample data when available. Done means compaction no longer fails with Arrow offset overflow and the regression is covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100