lance-format / lance-format/lance-c

Blob v2: how to read payload bytes from C?

Open
#76 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
5
Forks
14
Avg merge
3d 5h
Merged PRs (30d)
18

Description

I read Lance datasets from C through lance-c and am looking at blob columns.
Two questions, both about what is intended rather than what is broken.

What I see

lance-c v0.1.9 with lance at rev e934cc2c. Datasets written with pylance
11.0.0 at data_storage_version = "2.2"; the Blob v2 storage strategy is forced
through the field-metadata thresholds, so a small payload can stand in for a
large one.

A large_binary field tagged lance-encoding: blob (Blob v1): a projected scan
returns the payload bytes, and the stream schema keeps the metadata. Checked by
md5 on 1, 2 and 4 MiB values.

A Blob v2 field (Struct<data, uri> tagged
ARROW:extension:name = lance.blob.v2): a projected scan returns the
descriptor, for every strategy including inline. What arrives over the Arrow C
data interface is

format   = +s
metadata = (none)
children = [kind: C, position: L, size: L, blob_id: I, blob_uri: u]

kind follows the strategy (0 inline, 1 packed, 2 dedicated, matching the
threshold I set) and size is the true payload size.

1. Is a C entry point for blob payloads in scope?

lance.h has no blob functions, and the
liblance RFC does not
mention blobs in any of its four phases, so I cannot tell whether this is
deliberate or simply has not been asked for. The Rust crate lance-c depends on
has Dataset::take_blobs, read_blobs, read_blob_ranges and BlobFile at
the pinned rev.

From C I can read size and blob_uri out of the descriptor, which covers
inventory queries and, I assume, the external strategy. What I cannot do is
fetch the payload for the packed and dedicated strategies.

2. Is dropping ARROW:extension:name on the scan path intended?

lance_dataset_schema() carries ARROW:extension:name = lance.blob.v2 on the
field. The schema from lance_scanner_to_arrow_stream() does not: the
descriptor arrives as a bare five-field struct with no metadata at all. A
consumer that only sees the stream therefore cannot tell a blob descriptor from
an ordinary struct column, except by matching the child names.


If a C entry point for blob payloads is something you want, I am willing to
implement it. Tell me which shape you would accept — an opaque handle mirroring
BlobFile, or one planned batch read that returns Arrow for a row-id array —
and I will follow it, along with your error and lifetime conventions.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading lance.h and comparing lance_dataset_schema() with lance_scanner_to_arrow_stream(), then inspect the pinned Rust APIs Dataset::take_blobs, read_blobs, read_blob_ranges, and BlobFile. Done means the maintainers have selected an accepted C API shape for payload reads and clarified whether Blob v2 extension metadata should be preserved in scan results.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, rust
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.