lance-format / lance-format/lance

Harden fragment append against field-id schema mismatches

Open
#7,146 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

LanceOperation.Append currently maps incoming fragment columns by field id. This is correct, but it can produce a silent column scramble if the fragment metadata was generated from a schema whose field-id-to-name/type mapping differs from the target dataset.

A concrete failure mode was reported in #7035:

  1. A live dataset started with schema (id, name).
  2. A new column count was added with add_columns, so the live dataset field ids became:
    • id = 0
    • name = 1
    • count = 2
  3. A distributed writer generated fragments in an empty work directory using a canonical schema (id, count, name), so write_fragments assigned:
    • id = 0
    • count = 1
    • name = 2
  4. The coordinator copied the fragment data files into the live dataset and committed them with LanceOperation.Append.
  5. Append mapped by field id, so the fragment's count column was treated as the dataset's name column, and the fragment's name column was treated as the dataset's count column.

The resulting dataset failed later during reads with confusing low-level errors such as:

StringArray data should contain 2 buffers only (offsets and values)
left: 1
right: 2

This happened because the reader was asked to decode a physical int32 page as a string column.

The append behavior is contractually field-id based, so this is not a decoder bug. However, we could make this failure mode much easier to catch by validating incoming fragment schemas before appending. For example, when appending fragments, check that each field id maps to the same field name and compatible Arrow type in both the fragment schema and the target dataset schema. If they disagree, return a clear schema/fragment mismatch error instead of allowing a silently corrupted append.

Related: #7035

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at LanceOperation.Append and inspect how incoming fragment schemas are mapped against the target dataset schema, using the failure scenario in #7035 as context. Add validation for field-id name and compatible Arrow type mismatches, then verify that append returns a clear schema/fragment mismatch error instead of allowing corrupted data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.