lance-format / lance-format/lance
fix_schema renumbers a duplicated nested field without re-parenting its children, leaving the dataset unopenable
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
rust/lance/src/io/commit.rs::fix_schema runs on every commit whose fragments have more than one data file. When two files of one fragment claim the same field id, it maps the duplicate to a fresh id and applies the mapping to the schema with
let field = manifest.schema.mut_field_by_id(*old_field_id).unwrap();
field.id = *new_field_id;
Only id changes. The field's children keep parent_id = old_field_id, so for a nested field the committed manifest can no longer be read:
LanceError(Schema): Field 'item' (id=8) references parent id 2, which must appear earlier in the protobuf field list
Reproducer: a table with chunks: list<struct<text: utf8, ordinal: int32>> (field ids 2..5) stored in one file with the other columns; commit an Update { update_mode: RewriteColumns } whose updated fragment keeps that file unchanged and adds a second file with fields = [2, 3, 4, 5]. The commit succeeds and writes __lateral id=7 parent=-1, item id=8 parent=2, text id=9 parent=3, ordinal id=10 parent=3; every later open of the dataset fails with the error above. A flat column in the same situation is renumbered consistently and reads back, so the fixup is only broken for nested fields.
The writer is at fault for committing duplicate coverage (a writer of ours did), but a fixup that leaves the manifest unopenable is worse than refusing the commit. Either fix_schema should renumber the subtree with Field::set_id semantics (children re-parented), or the duplicate-coverage case should be rejected at validation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/io/commit.rs at fix_schema and trace how duplicate field IDs are remapped during commits. Reproduce the nested chunks/list/struct case described in the issue, then verify that the committed manifest can be opened and that nested parent IDs remain valid, or that duplicate coverage is rejected instead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100