lance-format / lance-format/lance-c
Blob v2: how to read payload bytes from C?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 5
- Forks
- 14
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 18
Description
I read Lance datasets from C through lance-c and am looking at blob columns.
Two questions, both about what is intended rather than what is broken.
What I see
lance-c v0.1.9 with lance at rev e934cc2c. Datasets written with pylance
11.0.0 at data_storage_version = "2.2"; the Blob v2 storage strategy is forced
through the field-metadata thresholds, so a small payload can stand in for a
large one.
A large_binary field tagged lance-encoding: blob (Blob v1): a projected scan
returns the payload bytes, and the stream schema keeps the metadata. Checked by
md5 on 1, 2 and 4 MiB values.
A Blob v2 field (Struct<data, uri> tagged
ARROW:extension:name = lance.blob.v2): a projected scan returns the
descriptor, for every strategy including inline. What arrives over the Arrow C
data interface is
format = +s
metadata = (none)
children = [kind: C, position: L, size: L, blob_id: I, blob_uri: u]
kind follows the strategy (0 inline, 1 packed, 2 dedicated, matching the
threshold I set) and size is the true payload size.
1. Is a C entry point for blob payloads in scope?
lance.h has no blob functions, and the
liblance RFC does not
mention blobs in any of its four phases, so I cannot tell whether this is
deliberate or simply has not been asked for. The Rust crate lance-c depends on
has Dataset::take_blobs, read_blobs, read_blob_ranges and BlobFile at
the pinned rev.
From C I can read size and blob_uri out of the descriptor, which covers
inventory queries and, I assume, the external strategy. What I cannot do is
fetch the payload for the packed and dedicated strategies.
2. Is dropping ARROW:extension:name on the scan path intended?
lance_dataset_schema() carries ARROW:extension:name = lance.blob.v2 on the
field. The schema from lance_scanner_to_arrow_stream() does not: the
descriptor arrives as a bare five-field struct with no metadata at all. A
consumer that only sees the stream therefore cannot tell a blob descriptor from
an ordinary struct column, except by matching the child names.
If a C entry point for blob payloads is something you want, I am willing to
implement it. Tell me which shape you would accept — an opaque handle mirroring
BlobFile, or one planned batch read that returns Arrow for a row-id array —
and I will follow it, along with your error and lifetime conventions.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading lance.h and comparing lance_dataset_schema() with lance_scanner_to_arrow_stream(), then inspect the pinned Rust APIs Dataset::take_blobs, read_blobs, read_blob_ranges, and BlobFile. Done means the maintainers have selected an accepted C API shape for payload reads and clarified whether Blob v2 extension metadata should be preserved in scan results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100