lance-format / lance-format/lance
lance.Dataset has no way to get the commit timestamp of the version it is holding
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Open a dataset at any version and nothing tells you when that version was committed:
ds = lance.dataset(uri, version=n)
ds.version # the int
# ...and no way to the timestamp
The value isn't missing, it's discarded. Dataset::version() builds
Version { version, timestamp, metadata } from the manifest that's already in memory -
no IO - and the binding keeps one field:
fn version(&self) -> PyResult<u64> {
Ok(self.ds.version().version)
}
metadata (the manifest summary: total_rows, total_fragments, total_files_size) goes the
same way.
The only thing in pylance that surfaces a timestamp at all is versions(), which lists and
reads every manifest in the table's history on every call, so you can filter down to the one
you were already holding. It also carries a // TODO: this API should support pagination.
The same field is dropped for other versions: read_version_transaction(v) returns
{version, timestamp, transaction}, but only the wrapper below it is bound:
pub async fn read_transaction_by_version(&self, version: u64) -> Result<Option<Transaction>> {
Ok(self.read_version_transaction(version).await?.transaction)
}
and reading the transaction already read that version's manifest.
Would exposing both be reasonable?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read Dataset::version() and the Python binding fn version() first; compare them with read_version_transaction(v) and versions() to trace the already-loaded timestamp and metadata. Done means the held version exposes its commit timestamp and manifest metadata without rereading table history, with the corresponding binding API available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100