lance-format / lance-format/lance
bug: VariableFullZipDecoder aborts the process on corrupt page metadata
@Xuanwo is already working on this.
Since Aug 28, 2026.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
VariableFullZipDecoder aborts the process on several corrupt-page shapes instead of returning an error. Some of these live on the item walk in unzip and are being fixed separately; the ones below all share a root cause the walk cannot address: a value read out of the page metadata is trusted before anything validates it against the bytes actually present.
1. drain indexes its row-start vectors with a row count the page bytes may not support
repdef_starts, data_starts, offset_starts and visible_item_counts get one entry per is_new_row control word plus a trailing sentinel, so their length is decided by the page bytes. drain indexes them at current_idx + num_rows, where num_rows comes from the scheduler. Nothing between unzip and drain compares the two.
With no control words, a 32-bit length prefix, num_rows = 2 and a page of [len=4u32]["abcd"], the page ends exactly on an item boundary, so no bounds check on the walk fires. unzip returns Ok with offset_starts == [0, 4], and drain(2) then evaluates offset_starts[2] and panics.
The reverse direction is silent rather than fatal: a page carrying more items than num_rows is accepted, and the surplus items sit in self.data and are never drained.
A strict offset_starts.len() == num_rows + 1 check would close both, but it needs a test over a rep-index sub-range read first, to confirm that every buffer handed to unzip really does carry a whole number of rows.
2. The row count drives allocation before it is validated
VariableFullZipDecoder::new preallocates four Vec::with_capacity(num_rows as usize + 1), and unzip computes bytes_cw, bytes_offsets and bytes_lengths by multiplying num_rows with no overflow check, before the walk validates anything. A page claiming an absurd row count aborts on capacity overflow or allocation failure, and on attempt to multiply with overflow in a debug build.
Every row costs at least one control word, or at least a length prefix when there are no control words. A single check that num_rows is not larger than the buffers could possibly carry would bound the allocation and also hand item 1 the invariant it needs.
3. bits_per_offset is truncated from the page protobuf before it is checked
FullZipScheduler reads layout.bits_per_offset as a u32 and narrows it with as u8 on the way to create_decoder. A page carrying 288 becomes 32 and 8224 becomes 32, so the value passes validation and the page decodes at the wrong width without any error. Validating layout.bits_per_offset while it is still a u32 is what closes this.
4. A panic! on a missing protobuf field
create_decoder has None => panic!("Full-zip layout must have a details field") for a full-zip layout with no details. That field comes from the file, so a corrupt page kills the process. create_decoder already returns Result.
Impact
All four are reachable from a corrupt or truncated file, and all four kill the process rather than failing the read. On a scan the panic surfaces through the decode task as an opaque aborted-task error, which hides the real cause. #6817 described the same diagnostic problem for the blob decoder.
Suggested grouping
Items 2, 3 and 4 are page-metadata validation and fit naturally in one change. Item 1 depends on the invariant item 2 would establish, so it is best done after, together with the sub-range test.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.