lance-format / lance-format/lance
data_stats() table size undercounts blob v2 data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
Blob v2 stores data in a different place so dataset.stats.data_stats() only counts the size of the pointer to it, not the full data. But when we calculate a table's size, usually (I think) we want to include the size of all the data involved.
Steps to reproduce
import pathlib, shutil
import pyarrow as pa
import lance
from lance.blob import BlobType, blob_field
URI = "/tmp/blob_size_mre"
PAYLOAD = 1 << 20 # 1 MiB -> past dedicated_size_threshold, so a sidecar
shutil.rmtree(URI, ignore_errors=True)
blob_type = BlobType()
field = blob_field("blob", dedicated_size_threshold=1024)
storage = pa.array(
[{"data": b"x" * PAYLOAD, "uri": None, "position": None, "size": None}],
type=blob_type.storage_type,
)
table = pa.table(
[pa.ExtensionArray.from_storage(blob_type, storage)], schema=pa.schema([field])
)
ds = lance.write_dataset(table, URI, data_storage_version="2.2")
reported = sum(f.bytes_on_disk for f in ds.stats.data_stats().fields)
on_disk = sum(p.stat().st_size for p in pathlib.Path(URI).rglob("*") if p.is_file())
print(f"payload written : {PAYLOAD:>9,}")
print(f"data_stats total : {reported:>9,}")
print(f"actual bytes used : {on_disk:>9,}")
Expected behavior
data_stats should return about 1Mb
Lance version
9.0.0
Language binding
Python
Environment
local osx
Logs / traceback
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Python reproduction and the data_stats() implementation, focusing on how Blob v2 sidecar files are accounted for. Compare reported bytes_on_disk with the total file sizes, then verify that data_stats() includes the full blob payload and reports about 1 MiB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100