Appending an all-null DictionaryArray fails with an out-of-bounds error

Description

Appending a valid record batch containing an all-null Dictionary<Int32, Utf8> column fails with an Arrow out-of-bounds error. Non-dictionary nullable columns work as expected.

The affected component is rust/lance-encoding, in PrimitiveStructuralEncoder::extract_validity_buf. The encoder removes the dictionary array's null buffer and rebuilds its ArrayData. For an all-null dictionary, the values dictionary is empty. Removing validity makes key 0 appear valid, and Arrow rejects it as an index into an empty dictionary.

This is related to #3962/#3974, but appears distinct: that issue concerned nullable input reaching dictionary encoding, while this reproducer fails while rebuilding an explicitly dictionary-typed all-null array.

Steps to reproduce

Use these dependencies:

[dependencies]
arrow = "58"
lance = { git = "https://github.com/lance-format/lance", tag = "v9.1.0-beta.8", default-features = false }
tokio = { version = "1.23", features = ["rt", "macros"] }

Run this program with cargo run:

use std::sync::Arc;

use arrow::array::{new_null_array, ArrayRef, RecordBatch, RecordBatchIterator, UInt64Array};
use arrow::datatypes::{DataType, Field, Schema};
use lance::dataset::{WriteMode, WriteParams};
use lance::Dataset;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let dictionary_type =
        DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8));

    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::UInt64, false),
        Field::new("value", dictionary_type.clone(), true),
    ]));

    let empty = RecordBatch::new_empty(schema.clone());
    let mut dataset = Dataset::write(
        RecordBatchIterator::new([Ok(empty)], schema.clone()),
        "memory://all_null_dictionary",
        Some(WriteParams {
            mode: WriteMode::Overwrite,
            ..Default::default()
        }),
    )
    .await
    .unwrap();

    let arrays: Vec<ArrayRef> = vec![
        Arc::new(UInt64Array::from(vec![1])),
        new_null_array(&dictionary_type, 1),
    ];
    let batch = RecordBatch::try_new(schema.clone(), arrays).unwrap();

    dataset
        .append(
            Box::new(RecordBatchIterator::new([Ok(batch)], schema)),
            None,
        )
        .await
        .unwrap();
}

Expected behavior

The append succeeds and reading the dataset returns one row whose dictionary column is null.

Lance version

v9.1.0-beta.8 (5107a99e3f3912851c8cbb3822bd5b4cbf7fda2f)

Language binding

Rust

Environment

Fedora Linux, x86_64, memory:// storage, Rust 1.96.0

Logs / traceback

LanceError(Arrow): Invalid argument error: Value at position 0 out of bounds:
0 (should be in [0, -1])

Run the provided Rust reproducer with the listed Lance and Arrow dependencies, then inspect rust/lance-encoding at PrimitiveStructuralEncoder::extract_validity_buf. Trace how the dictionary array's validity buffer and empty values dictionary are rebuilt, and verify the fix by confirming append succeeds and reading returns one row with a null dictionary column.