lance-format / lance-format/lance

bug(overlay): struct cell validity is not replaced by a data overlay

Open
#9,077 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Summary

Data overlay resolution cannot change the validity (null-ness) of a struct cell. Overlaying a nullable struct's children updates the child values but silently preserves the base struct's null buffer, so NULL -> non-NULL and non-NULL -> NULL transitions are lost.

This contradicts the overlay contract, which states that a covered offset holding NULL overrides that cell to NULL (see rust/lance/src/dataset/overlay/writer.rs module docs and docs/src/format/table/data_overlay_file.md).

Cause

Overlay resolution is per atomic field, and a struct is recursed through rather than treated as an atomic unit — only its leaves are atomic (rust/lance/src/dataset/overlay.rs:371-379).

At merge time, splice_by_ids destructures the base StructArray and rebuilds it with the base null buffer:

// rust/lance/src/dataset/overlay.rs:566-575
let (fields, mut children, nulls) = structs.clone().into_parts();
children[child_pos] = splice_by_ids(...)?;
Ok(Arc::new(StructArray::try_new_with_length(
    fields, children, nulls, len,   // <- base `nulls`, never the overlay's
)?))

The writer does faithfully encode the struct's validity into the overlay value file; the read path structurally ignores it.

Reproduction

  1. Create a two-row dataset with a nullable struct column, child values [1, 2], struct validity [false, true].
  2. Overlay both rows with child values [10, 20] and struct validity [true, false].
  3. Commit and scan.

Observed: child values [10, 20] (correct), struct validity [false, true] (stale).
Expected: struct validity [true, false].

Options

  1. Represent and merge validity at every struct ancestor — either give struct parents their own atomic-field entry carrying only validity, or have the merge recompute parent nulls from covered children. Read-path change, possibly a format change.
  2. Reject nullable struct ancestors at the writer until (1) lands — fail closed rather than returning wrong data.

Interim mitigation

Tracking this separately from https://github.com/lance-format/lance/pull/8761, which adds OverlayWriter. That PR takes approach (2) as a stopgap so the writer cannot produce an overlay whose semantics the read path will not honor. This issue tracks doing it properly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read rust/lance/src/dataset/overlay.rs, especially atomic-field resolution and splice_by_ids around lines 371-379 and 566-575, alongside the overlay contract in rust/lance/src/dataset/overlay/writer.rs and docs/src/format/table/data_overlay_file.md. Determine how struct validity should be represented or merged, then verify that the reproduction changes validity from [false, true] to [true, false] after scanning.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.