lance-format / lance-format/lance

fix_schema renumbers a duplicated nested field without re-parenting its children, leaving the dataset unopenable

Open
#9,424 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

rust/lance/src/io/commit.rs::fix_schema runs on every commit whose fragments have more than one data file. When two files of one fragment claim the same field id, it maps the duplicate to a fresh id and applies the mapping to the schema with

let field = manifest.schema.mut_field_by_id(*old_field_id).unwrap();
field.id = *new_field_id;

Only id changes. The field's children keep parent_id = old_field_id, so for a nested field the committed manifest can no longer be read:

LanceError(Schema): Field 'item' (id=8) references parent id 2, which must appear earlier in the protobuf field list

Reproducer: a table with chunks: list<struct<text: utf8, ordinal: int32>> (field ids 2..5) stored in one file with the other columns; commit an Update { update_mode: RewriteColumns } whose updated fragment keeps that file unchanged and adds a second file with fields = [2, 3, 4, 5]. The commit succeeds and writes __lateral id=7 parent=-1, item id=8 parent=2, text id=9 parent=3, ordinal id=10 parent=3; every later open of the dataset fails with the error above. A flat column in the same situation is renumbered consistently and reads back, so the fixup is only broken for nested fields.

The writer is at fault for committing duplicate coverage (a writer of ours did), but a fixup that leaves the manifest unopenable is worse than refusing the commit. Either fix_schema should renumber the subtree with Field::set_id semantics (children re-parented), or the duplicate-coverage case should be rejected at validation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance/src/io/commit.rs at fix_schema and trace how duplicate field IDs are remapped during commits. Reproduce the nested chunks/list/struct case described in the issue, then verify that the committed manifest can be opened and that nested parent IDs remain valid, or that duplicate coverage is rejected instead.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.