lance-format / lance-format/lance
Harden fragment append against field-id schema mismatches
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
LanceOperation.Append currently maps incoming fragment columns by field id. This is correct, but it can produce a silent column scramble if the fragment metadata was generated from a schema whose field-id-to-name/type mapping differs from the target dataset.
A concrete failure mode was reported in #7035:
- A live dataset started with schema
(id, name). - A new column
countwas added withadd_columns, so the live dataset field ids became:id = 0name = 1count = 2
- A distributed writer generated fragments in an empty work directory using a canonical schema
(id, count, name), sowrite_fragmentsassigned:id = 0count = 1name = 2
- The coordinator copied the fragment data files into the live dataset and committed them with
LanceOperation.Append. - Append mapped by field id, so the fragment's
countcolumn was treated as the dataset'snamecolumn, and the fragment'snamecolumn was treated as the dataset'scountcolumn.
The resulting dataset failed later during reads with confusing low-level errors such as:
StringArray data should contain 2 buffers only (offsets and values)
left: 1
right: 2
This happened because the reader was asked to decode a physical int32 page as a string column.
The append behavior is contractually field-id based, so this is not a decoder bug. However, we could make this failure mode much easier to catch by validating incoming fragment schemas before appending. For example, when appending fragments, check that each field id maps to the same field name and compatible Arrow type in both the fragment schema and the target dataset schema. If they disagree, return a clear schema/fragment mismatch error instead of allowing a silently corrupted append.
Related: #7035
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at LanceOperation.Append and inspect how incoming fragment schemas are mapped against the target dataset schema, using the failure scenario in #7035 as context. Add validation for field-id name and compatible Arrow type mismatches, then verify that append returns a clear schema/fragment mismatch error instead of allowing corrupted data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100