lance-format / lance-format/lance

data_stats() table size undercounts blob v2 data

Open
#8,193 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

Blob v2 stores data in a different place so dataset.stats.data_stats() only counts the size of the pointer to it, not the full data. But when we calculate a table's size, usually (I think) we want to include the size of all the data involved.

Steps to reproduce
import pathlib, shutil
import pyarrow as pa
import lance
from lance.blob import BlobType, blob_field

URI = "/tmp/blob_size_mre"
PAYLOAD = 1 << 20  # 1 MiB -> past dedicated_size_threshold, so a sidecar

shutil.rmtree(URI, ignore_errors=True)

blob_type = BlobType()
field = blob_field("blob", dedicated_size_threshold=1024)
storage = pa.array(
    [{"data": b"x" * PAYLOAD, "uri": None, "position": None, "size": None}],
    type=blob_type.storage_type,
)
table = pa.table(
    [pa.ExtensionArray.from_storage(blob_type, storage)], schema=pa.schema([field])
)
ds = lance.write_dataset(table, URI, data_storage_version="2.2")

reported = sum(f.bytes_on_disk for f in ds.stats.data_stats().fields)
on_disk = sum(p.stat().st_size for p in pathlib.Path(URI).rglob("*") if p.is_file())
print(f"payload written   : {PAYLOAD:>9,}")
print(f"data_stats total  : {reported:>9,}")
print(f"actual bytes used : {on_disk:>9,}")
Expected behavior

data_stats should return about 1Mb

Lance version

9.0.0

Language binding

Python

Environment

local osx

Logs / traceback

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Python reproduction and the data_stats() implementation, focusing on how Blob v2 sidecar files are accounted for. Compare reported bytes_on_disk with the total file sizes, then verify that data_stats() includes the full blob payload and reports about 1 MiB.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.