bug: struct-nested blob v2 column cannot be compacted — "Blob struct missing `data` field" on any fragment rewrite

Description

A dataset holding a struct-nested blob v2 column cannot be compacted once any fragment needs rewriting. compact_files() fails with:

OSError: Invalid user input: Blob struct missing `data` field,
/rust/lance/src/dataset/blob.rs:836:28

The failure is loud and there is no data loss, but the dataset is permanently un-compactable: it can never reclaim deleted-row space or merge small fragments. For a table that accumulates deletes, that is unbounded space growth with no remedy.

The trigger is the struct nesting, not blob v2 and not the payload values. A top-level blob v2 column with the identical payloads — including b"" and NULL interleaved — compacts cleanly on the same pin.

blob.rs:836 appears to inspect the top-level column struct (here {mime, payload}) as if it were the blob storage struct (which is where data lives), rather than descending to the nested blob leaf.

Steps to reproduce

Pure pylance, no other dependency. Four rows, one delete.

import tempfile
import lance
import pyarrow as pa

NESTED = pa.struct([pa.field("mime", pa.string()), lance.blob_field("payload")])
SCHEMA = pa.schema([pa.field("id", pa.int64()), pa.field("asset", NESTED)])
VALS = [b"p0", b"", None, b"p3"]

uri = f"{tempfile.mkdtemp()}/t.lance"
arr = pa.StructArray.from_arrays(
    [pa.array(["image/png"] * 4), lance.blob_array(VALS)],
    fields=[pa.field("mime", pa.string()), lance.blob_field("payload")],
)
ds = lance.write_dataset(
    pa.Table.from_arrays([pa.array(range(4), pa.int64()), arr], schema=SCHEMA),
    uri,
    data_storage_version="2.2",
)
ds.delete("id % 2 = 0")
lance.dataset(uri).optimize.compact_files()   # raises

Scope

Sweeping nesting against the operation sequence isolates it exactly. Observed on pylance 10.0.0-beta.7:

nested struct<mime, blob>  write only    : compacted OK
nested struct<mime, blob>  write+delete  : OSError: Blob struct missing `data` field
top-level blob             write only    : compacted OK
top-level blob             write+delete  : compacted OK

• Compaction succeeds on a nested column when no fragment needs rewriting (write-only). Any rewrite trigger fails — a delete alone is sufficient; an append is not required.
• A top-level blob v2 column is unaffected, with the same [b"p0", b"", None, b"p3"] payloads. So this is not the null/empty descriptor family (lance-format/lance#7955 / lance-format/lance#8067 / lance-format/lance#8069) — those payload shapes compact fine here when the column is top-level.
• Reproduces identically on 10.0.0-beta.6 and 10.0.0-beta.7, so it is not a regression from lance-format/lance#8070 and was not fixed by it.
• Not API misuse: the same nested schema reads and writes correctly; only compaction fails.

Why this shape matters

struct<..., blob> is the natural way to carry a payload alongside its metadata (image.image_bytes next to an error/mime field), and it is the shape used in practice by callers who need per-row provenance beside the bytes. Those tables are exactly the ones that accumulate deletes and most need compaction.

Expected

A dataset with a struct-nested blob v2 column compacts successfully, with surviving rows keeping their exact payloads — including b"" preserved as empty and NULL preserved as null.

Environment

• pylance 10.0.0-beta.7 (and 10.0.0-beta.6)
• Python 3.12
• macOS arm64

Found by an adversarial blob read/write test suite; no fault injection is involved in the reproduction — the triggering operations are plain write_dataset, delete and compact_files.