lance-format / lance-format/lance

lance.Dataset has no way to get the commit timestamp of the version it is holding

Open
#8,928 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Open a dataset at any version and nothing tells you when that version was committed:

ds = lance.dataset(uri, version=n)
ds.version        # the int
                  # ...and no way to the timestamp

The value isn't missing, it's discarded. Dataset::version() builds
Version { version, timestamp, metadata } from the manifest that's already in memory -
no IO - and the binding keeps one field:

fn version(&self) -> PyResult<u64> {
    Ok(self.ds.version().version)
}

metadata (the manifest summary: total_rows, total_fragments, total_files_size) goes the
same way.

The only thing in pylance that surfaces a timestamp at all is versions(), which lists and
reads every manifest in the table's history on every call, so you can filter down to the one
you were already holding. It also carries a // TODO: this API should support pagination.

The same field is dropped for other versions: read_version_transaction(v) returns
{version, timestamp, transaction}, but only the wrapper below it is bound:

pub async fn read_transaction_by_version(&self, version: u64) -> Result<Option<Transaction>> {
    Ok(self.read_version_transaction(version).await?.transaction)
}

and reading the transaction already read that version's manifest.

Would exposing both be reasonable?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read Dataset::version() and the Python binding fn version() first; compare them with read_version_transaction(v) and versions() to trace the already-loaded timestamp and metadata. Done means the held version exposes its commit timestamp and manifest metadata without rereading table history, with the corresponding binding API available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend-api-design, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.