lance-format / lance-format/lance
§5.3 Row ID sequences spill to data files as a hidden column
@BubbleCal is already working on this.
Since Sep 15, 2026.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
See: https://docs.google.com/document/d/1x6Xon1XNZ5XaXHekk5kU0Ujg0A9FCOSLAP1Ry9XTMx8/edit?tab=t.0
Row IDs, created-at, and last-updated-at cannot always live in the manifest. Inlining is a good optimization while they are a few bytes; past that they need to be in separate files. Rather than a new file type, store them as special columns in the data files, reusing existing encoding and file-location infrastructure, at fixed field IDs:
_rowid->-3_row_created_at_version->-4_row_last_updated_at_version->-5
Field ID is an i32; non-negative values are user fields, negatives are system fields (-1 and -2 are already in use). This likely requires a spec change to reserve all negative IDs for system use, though it is consistent with how the libraries already behave.
Placement depends on the rebase behavior needed. On a commit conflict we must (a) re-mint new row IDs, (b) re-mint _row_created_at_version, (c) recompute _row_last_updated_at_version — the first two only when the write adds rows. Rewriting inline metadata is fastest (in-memory); a dedicated data file is next (minimal write amplification); rewriting a main data file is expensive and should be avoided.
| Operation | Placement |
|---|---|
| Append | all three trivially inline |
| Update (RewriteRows) | _rowid, _row_created_at_version inline or in the main data file; _row_last_updated_at_version trivially inline |
| Mixed updates + appends | all three in a dedicated data file |
| Compaction | all retry-stable, so directly in a data file |
T
This replaces the unused RowIdMeta::External stub. A new feature flag gates spilled row IDs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.