lance-format / lance-format/lance
update LanceNamespace::get_table_stats() to use manifests instead of file metadata
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We have two paths to calculate "the size of a table":
- lance_table::format::Manifest::summary().total_files_size, which uses each file's file_size_bytes (recorded in the manifest, basically free)
- lance::dataset::statistics::DatasetStatisticsExt::calculate_data_stats() which is what feeds LanceDB's TableStatistics.total_bytes, which reads metadata from each file.
Path 1 is O(1) and Path 2 is O(data files). But Path 2 also calculates per-field stats. So if you need stats for each field, it's the only way to do it, but otherwise callers should use the manifest. One caller that still uses Path 2 but just sums up each field after is https://github.com/lance-format/lance/blob/acd623558faf642094818a63f37a2eafa9756a4d/rust/lance-namespace-impls/src/dir.rs#L4826-L4840, we should move that to use the manifest instead.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-namespace-impls/src/dir.rs at the table-statistics caller around lines 4826-4840. Compare its use of DatasetStatisticsExt::calculate_data_stats() with lance_table::format::Manifest::summary().total_files_size. Done means this caller obtains total table bytes from the manifest while retaining per-field statistics only where they are needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100