lance-format / lance-format/lance

bug: arrow offset overflow

Open
#9,223 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

I'm seeing some compactions fail with LanceError(Arrow): Offset overflow error: N, where N is always just over i32::MAX (observed ~ 2,150M). I've seen it come from 2 sites, both in the decode path where arrow_select::concat::concat stitches decoded pages into a single Arrow array: CompositeDecodeTask::decode at lance-encoding/src/array_encoding/logical/struct.rs:594 and StructuralCompositeDecodeArrayTask::decode at lance-encoding/src/encodings/logical/primitive.rs:4501 (and :4367). When one compaction's input carries more than 2 GiB of variable-width data in a single i32-offset column (Utf8/Binary/List), the concat overflows and the task fails. I've seen this on both lance 11.0.0-beta.22 and 12.0.0-beta.16;

Steps to reproduce
I can provide sample data that reproduces this, but it's too big to share here immediately.
Expected behavior

No response

Lance version

11.0.0-beta.22

Language binding

Rust

Environment

No response

Logs / traceback

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lance-encoding/src/array_encoding/logical/struct.rs at CompositeDecodeTask::decode:594 and lance-encoding/src/encodings/logical/primitive.rs at StructuralCompositeDecodeArrayTask::decode:4367 and :4501. Reproduce with variable-width Utf8, Binary, or List data exceeding 2 GiB in one i32-offset column, using the reporter's sample data when available. Done means compaction no longer fails with Arrow offset overflow and the regression is covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.