lance-format / lance-format/lance

debug_assert!(preamble_action == PreambleAction::Take) fires in DecodeMiniBlockTask::map_range on a multi-row take over a 2.1 list column whose rows each span multiple mini-block chunks (3.0.1 and 10.0.0)

Open
#8,558 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

A debug build panics at
lance-encoding/src/encodings/logical/primitive.rs in
DecodeMiniBlockTask::map_range, at the debug_assert!(preamble_action == PreambleAction::Take) inside the empty-row-range arm ("We hit this case when
all we needed was the preamble"):

assertion failed: preamble_action == PreambleAction::Take
lance-encoding-3.0.1/src/encodings/logical/primitive.rs:306:17   (our pin)
lance-encoding-10.0.0/src/encodings/logical/primitive.rs:313:17  (current release)

The same assertion is present on main today (line 326). We found no existing
issue matching it (searched PreambleAction, preamble_action map_range).

Minimal reproducer

Dataset with one List<Utf8> column, data storage version 2.1: 3 rows, each
a list of 512 Utf8 items of 16 bytes
, then dataset.take(&[0, 2], schema).
Debug build panics; the panic is deterministic.

use std::sync::Arc;
use arrow_array::builder::{ListBuilder, StringBuilder};
use arrow_array::{ArrayRef, Int32Array, RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
use lance::dataset::{Dataset, WriteParams};
use lance_file::version::LanceFileVersion;

#[tokio::main]
async fn main() {
    let n_rows = 3;
    let items = 512;
    let ids = Int32Array::from_iter_values(0..n_rows as i32);
    let mut b = ListBuilder::new(StringBuilder::new());
    for row in 0..n_rows {
        for idx in 0..items {
            b.values().append_value(format!("r{row:03}i{idx:06}xaaaa"));
        }
        b.append(true);
    }
    let refs: ArrayRef = Arc::new(b.finish());
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("refs", refs.data_type().clone(), true),
    ]));
    let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(ids), refs]).unwrap();
    let reader = RecordBatchIterator::new([Ok(batch)], schema);
    let params = WriteParams {
        data_storage_version: Some(LanceFileVersion::V2_1),
        ..Default::default()
    };
    Dataset::write(reader, "./repro-ds", Some(params)).await.unwrap();
    let dataset = Dataset::open("./repro-ds").await.unwrap();
    let projected = dataset.schema().clone();
    // Debug build: panics at the map_range debug_assert.
    let out = dataset.take(&[0, 2], projected).await.unwrap();
    println!("rows={}", out.num_rows());
}

Trigger boundary (from a 144-case automated search, run twice with

identical results)

  • Every row of the list column must span multiple mini-block chunks: 512x16B
    and 384x16B items fire; 320x16B and 256x16B do not (the boundary tracks the
    4KiB chunk edge).
  • The take must contain two or more rows ([0,2], [1,7], [3,17,31],
    full ranges). Single-row takes never fire.
  • Irrelevant: List vs LargeList, item-field nullability, a sibling large
    Utf8 column, deletion vectors, row count beyond 3, taking only 2 rows of a
    2-row dataset (passes — a row must exist beyond the taken window).
  • A single giant row among small rows does NOT fire; the uniform
    rows-all-spanning-chunks shape is required.

Release behavior: no panic, output verified byte-correct (3.0.1 and 10.0.0)

In release the debug_assert compiles out and the empty-range arm returns
(0..items_in_preamble, 0..first_row_start). We wrote a release-build
verifier that validates every id, list length, and item string of the take
result against the written data across all triggering shapes (including a full
32-row take and 4096-item rows): all byte-correct at both 3.0.1 and
10.0.0
. So we are
reporting an invariant/scheduler-model mismatch, not corruption or data loss —
though we cannot judge whether the scheduler state that trips the assert is
benign by design or accidentally-safe.

Question

Is the scheduler's empty-row-range request with a non-Take preamble action
legal (assertion too strict), or is it a real scheduling bug that release
survives by accident? PR #4823 (all-preamble chunk scheduling) predates 3.0.1
and addresses the adjacent hard-assert arm, not this one.

Environment

  • lance / lance-file 3.0.1 with arrow 57.3.0, and lance / lance-file 10.0.0
    with arrow 58.4.0 — identical behavior on both.
  • rustc 1.95.0 (59807616e 2026-04-14), macOS arm64 (Darwin 24.3.0).
  • Full debug backtrace (3.0.1): panic in DecodeMiniBlockTask::map_range
    (frame 3), reached from Dataset::take via
    StructuralListDecodeTask/RepDefStructDecodeTask decode and
    lance/src/io/exec/take.rs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lance-encoding/src/encodings/logical/primitive.rs at DecodeMiniBlockTask::map_range, then trace the request from Dataset::take through lance/src/io/exec/take.rs and the structural list decode tasks. Run the supplied multi-row List reproducer in a debug build; done means establishing whether the non-Take preamble action is legal and correcting the assertion or scheduling logic with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.