Implement Blob v2 exact sharing for ingested blobs

Problem

Blob v2 can already choose among inline, packed, dedicated, and external placement, but the write path still treats each ingested blob value as an independent physical object. This makes workloads with repeated source blobs store the same bytes multiple times even when the caller can identify that multiple rows refer to the exact same source value.

Discussion #7174 proposes an object-layer direction for Blob v2. This issue tracks the first narrow implementation step: exact sharing for ingested blobs.

Scope

Add write-time exact sharing for blob values that are ingested into Lance-managed storage.

The sharing identity should be provided as a write-time source identity, scoped to a single write / data file. It must not become a dataset-global object id, and it must not create cross-table blob ownership or lifecycle dependencies.

Out of scope for this issue:

• delta blob encoding
• chunked blob encoding
• video or domain-specific blob packs
• cross-table blob sharing
• a public plugin system for blob object backends

Expected behavior

When multiple rows in the same write provide the same source identity for an ingested blob, Lance should store the physical bytes once and write descriptors that resolve to the same logical bytes.

Existing callers that do not provide a source identity should keep the current behavior.

Compatibility requirements

• Existing Blob v2 datasets remain readable.
• The default public read schema remains compatible with existing Blob v2 descriptor behavior.
• Any helper fields used to carry the write-time source identity must not leak into ordinary read schemas or persisted logical schemas.
• The implementation must fail clearly if exact sharing cannot be applied safely.

Acceptance criteria

• Rust write path supports write-local exact sharing for ingested Blob v2 values.
• Python write path can pass or infer the source identity for ingested external blob values.
• Tests cover duplicate source identity, distinct source identity, missing source identity, and read-back correctness.
• Tests cover that ordinary read schemas do not expose the write-time helper identity.
• Documentation or examples explain the feature boundary and explicitly distinguish it from cross-table sharing.

Related

• https://github.com/lance-format/lance/discussions/7174
• https://github.com/lance-format/lance/issues/4947
• https://github.com/lance-format/lance/issues/6372

Start with the Rust write path for Blob v2 and the Python write path for ingested external blobs. Trace how source identity is carried during one write and verify it is absent from ordinary read and persisted logical schemas. Add coverage for duplicate, distinct, and missing identities plus read-back correctness, and document the write-local boundary versus cross-table sharing.