lance-format / lance-format/lance

bug: VariableFullZipDecoder aborts the process on corrupt page metadata

Open
#8,768 0 comments 0 reactions 1 assignee View on GitHub

@Xuanwo is already working on this.

Since Aug 28, 2026.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

VariableFullZipDecoder aborts the process on several corrupt-page shapes instead of returning an error. Some of these live on the item walk in unzip and are being fixed separately; the ones below all share a root cause the walk cannot address: a value read out of the page metadata is trusted before anything validates it against the bytes actually present.

1. drain indexes its row-start vectors with a row count the page bytes may not support

repdef_starts, data_starts, offset_starts and visible_item_counts get one entry per is_new_row control word plus a trailing sentinel, so their length is decided by the page bytes. drain indexes them at current_idx + num_rows, where num_rows comes from the scheduler. Nothing between unzip and drain compares the two.

With no control words, a 32-bit length prefix, num_rows = 2 and a page of [len=4u32]["abcd"], the page ends exactly on an item boundary, so no bounds check on the walk fires. unzip returns Ok with offset_starts == [0, 4], and drain(2) then evaluates offset_starts[2] and panics.

The reverse direction is silent rather than fatal: a page carrying more items than num_rows is accepted, and the surplus items sit in self.data and are never drained.

A strict offset_starts.len() == num_rows + 1 check would close both, but it needs a test over a rep-index sub-range read first, to confirm that every buffer handed to unzip really does carry a whole number of rows.

2. The row count drives allocation before it is validated

VariableFullZipDecoder::new preallocates four Vec::with_capacity(num_rows as usize + 1), and unzip computes bytes_cw, bytes_offsets and bytes_lengths by multiplying num_rows with no overflow check, before the walk validates anything. A page claiming an absurd row count aborts on capacity overflow or allocation failure, and on attempt to multiply with overflow in a debug build.

Every row costs at least one control word, or at least a length prefix when there are no control words. A single check that num_rows is not larger than the buffers could possibly carry would bound the allocation and also hand item 1 the invariant it needs.

3. bits_per_offset is truncated from the page protobuf before it is checked

FullZipScheduler reads layout.bits_per_offset as a u32 and narrows it with as u8 on the way to create_decoder. A page carrying 288 becomes 32 and 8224 becomes 32, so the value passes validation and the page decodes at the wrong width without any error. Validating layout.bits_per_offset while it is still a u32 is what closes this.

4. A panic! on a missing protobuf field

create_decoder has None => panic!("Full-zip layout must have a details field") for a full-zip layout with no details. That field comes from the file, so a corrupt page kills the process. create_decoder already returns Result.

Impact

All four are reachable from a corrupt or truncated file, and all four kill the process rather than failing the read. On a scan the panic surfaces through the decode task as an opaque aborted-task error, which hides the real cause. #6817 described the same diagnostic problem for the blob decoder.

Suggested grouping

Items 2, 3 and 4 are page-metadata validation and fit naturally in one change. Item 1 depends on the invariant item 2 would establish, so it is best done after, together with the sub-range test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.