lance-format / lance-format/lance
Support version metadata when writing a dataset
Open
Nobody has claimed this yet.
A-python
enhancement
help wanted
rust
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Motivation
Version metadata can be used for debugging and lineage tracking
Proto definition
- Deprecate https://github.com/lancedb/lance/blob/cb092db625baade0b5568c1a98039e5a991d48e8/protos/table.proto#L134-L139
- Make it an optional field instead
Write
- Add Rust struct for VersionAuxData (https://github.com/eto-ai/lance/blob/main/protos/format.proto#L79) along with pb<>rust conversion traits
- Write the aux data section similar to how index metadata is written (https://github.com/eto-ai/lance/blob/main/rust/src/io/writer.rs#L79).
write_manifestshould take additional version metadata as parameter
note make sure the version_aux_data file position is tested
Read
- Add fn to read version aux data in the Version struct (https://github.com/eto-ai/lance/blob/main/rust/src/dataset.rs#L66)
- Instead of returning a PyDict for each version (https://github.com/eto-ai/lance/blob/main/python/src/dataset.rs#L182), create a pyo3 struct Version that is exposed to python. This struct should have a function to read the aux data and cache it in memory.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with protos/table.proto and protos/format.proto, then inspect Rust's existing index metadata writing in rust/src/io/writer.rs. Trace Version in rust/src/dataset.rs and the current Python version representation in python/src/dataset.rs. Done means version auxiliary data is written and readable through the exposed Python Version, with its file position covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100