lance-format / lance-format/lance

add_columns should support reading existing Blob v2 columns in UDFs

Open
#7,168 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-encoding enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

Dataset.add_columns(..., read_columns=["blob"]) fails when blob is an existing Blob v2 column.

This affects UDF-based schema evolution where the new column is derived from an existing Blob v2 descriptor column.

Reproduction

import pyarrow as pa
import lance

values = [
    b"inline",
    b"p" * (64 * 1024 + 1024),
    b"d" * (4 * 1024 * 1024 + 1024),
    external_blob.as_uri(),
]

ds = lance.write_dataset(
    pa.table({"id": range(4), "blob": lance.blob_array(values)}),
    uri,
    data_storage_version="2.2",
    initial_bases=[
        lance.DatasetBasePath(external_base.as_uri(), name="external", id=1)
    ],
)

@lance.batch_udf(output_schema=pa.schema([pa.field("blob_kind", pa.int32())]))
def blob_kind(batch):
    return pa.record_batch([batch["blob"].field("kind")], ["blob_kind"])

ds.add_columns(blob_kind, read_columns=["blob"])

Error

OSError: Invalid user input: there were more fields in the schema than provided column indices / infos,
rust/lance-encoding/src/decoder.rs:454:13

Expected behavior

add_columns should be able to read an existing Blob v2 column as a descriptor struct when it is listed in read_columns.

The UDF should receive the Blob v2 descriptor batch, and derived columns should be written successfully.

Notes

This is separate from writing new Blob v2 columns through add_columns.

The existing Blob v2 add_columns tests cover writing new Blob v2 values through RecordBatchReader and BatchUDF, including inline, packed, dedicated, and external. This issue is about reading an existing Blob v2 column during the UDF input scan.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Dataset.add_columns and batch_udf entry points, then inspect the UDF input scan around the decoder error in rust/lance-encoding/src/decoder.rs:454. Run the provided reproduction and review the existing Blob v2 add_columns tests for related coverage. Done means an existing Blob v2 column is delivered to the UDF as its descriptor struct and the derived column writes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.