lance-format / lance-format/lance
debug_assert!(preamble_action == PreambleAction::Take) fires in DecodeMiniBlockTask::map_range on a multi-row take over a 2.1 list column whose rows each span multiple mini-block chunks (3.0.1 and 10.0.0)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
A debug build panics at
lance-encoding/src/encodings/logical/primitive.rs in
DecodeMiniBlockTask::map_range, at the debug_assert!(preamble_action == PreambleAction::Take) inside the empty-row-range arm ("We hit this case when
all we needed was the preamble"):
assertion failed: preamble_action == PreambleAction::Take
lance-encoding-3.0.1/src/encodings/logical/primitive.rs:306:17 (our pin)
lance-encoding-10.0.0/src/encodings/logical/primitive.rs:313:17 (current release)
The same assertion is present on main today (line 326). We found no existing
issue matching it (searched PreambleAction, preamble_action map_range).
Minimal reproducer
Dataset with one List<Utf8> column, data storage version 2.1: 3 rows, each
a list of 512 Utf8 items of 16 bytes, then dataset.take(&[0, 2], schema).
Debug build panics; the panic is deterministic.
use std::sync::Arc;
use arrow_array::builder::{ListBuilder, StringBuilder};
use arrow_array::{ArrayRef, Int32Array, RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
use lance::dataset::{Dataset, WriteParams};
use lance_file::version::LanceFileVersion;
#[tokio::main]
async fn main() {
let n_rows = 3;
let items = 512;
let ids = Int32Array::from_iter_values(0..n_rows as i32);
let mut b = ListBuilder::new(StringBuilder::new());
for row in 0..n_rows {
for idx in 0..items {
b.values().append_value(format!("r{row:03}i{idx:06}xaaaa"));
}
b.append(true);
}
let refs: ArrayRef = Arc::new(b.finish());
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("refs", refs.data_type().clone(), true),
]));
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(ids), refs]).unwrap();
let reader = RecordBatchIterator::new([Ok(batch)], schema);
let params = WriteParams {
data_storage_version: Some(LanceFileVersion::V2_1),
..Default::default()
};
Dataset::write(reader, "./repro-ds", Some(params)).await.unwrap();
let dataset = Dataset::open("./repro-ds").await.unwrap();
let projected = dataset.schema().clone();
// Debug build: panics at the map_range debug_assert.
let out = dataset.take(&[0, 2], projected).await.unwrap();
println!("rows={}", out.num_rows());
}
Trigger boundary (from a 144-case automated search, run twice with
identical results)
- Every row of the list column must span multiple mini-block chunks: 512x16B
and 384x16B items fire; 320x16B and 256x16B do not (the boundary tracks the
4KiB chunk edge). - The
takemust contain two or more rows ([0,2],[1,7],[3,17,31],
full ranges). Single-row takes never fire. - Irrelevant:
ListvsLargeList, item-field nullability, a sibling large
Utf8 column, deletion vectors, row count beyond 3, taking only 2 rows of a
2-row dataset (passes — a row must exist beyond the taken window). - A single giant row among small rows does NOT fire; the uniform
rows-all-spanning-chunks shape is required.
Release behavior: no panic, output verified byte-correct (3.0.1 and 10.0.0)
In release the debug_assert compiles out and the empty-range arm returns
(0..items_in_preamble, 0..first_row_start). We wrote a release-build
verifier that validates every id, list length, and item string of the take
result against the written data across all triggering shapes (including a full
32-row take and 4096-item rows): all byte-correct at both 3.0.1 and
10.0.0. So we are
reporting an invariant/scheduler-model mismatch, not corruption or data loss —
though we cannot judge whether the scheduler state that trips the assert is
benign by design or accidentally-safe.
Question
Is the scheduler's empty-row-range request with a non-Take preamble action
legal (assertion too strict), or is it a real scheduling bug that release
survives by accident? PR #4823 (all-preamble chunk scheduling) predates 3.0.1
and addresses the adjacent hard-assert arm, not this one.
Environment
- lance / lance-file 3.0.1 with arrow 57.3.0, and lance / lance-file 10.0.0
with arrow 58.4.0 — identical behavior on both. - rustc 1.95.0 (59807616e 2026-04-14), macOS arm64 (Darwin 24.3.0).
- Full debug backtrace (3.0.1): panic in
DecodeMiniBlockTask::map_range
(frame 3), reached fromDataset::takevia
StructuralListDecodeTask/RepDefStructDecodeTaskdecode and
lance/src/io/exec/take.rs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lance-encoding/src/encodings/logical/primitive.rs at DecodeMiniBlockTask::map_range, then trace the request from Dataset::take through lance/src/io/exec/take.rs and the structural list decode tasks. Run the supplied multi-row List reproducer in a debug build; done means establishing whether the non-Take preamble action is legal and correcting the assertion or scheduling logic with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100