lance-format / lance-format/lance
The fields of the struct must include the root field name when filling in columns in to_batches
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
iter_ds = ds.to_batches(
batch_size=batch_size,
columns=['numeric_feature', 'payload', 'metadata', 'metadata.quality_score'],
scan_in_order=False,
)
Among them, metadata is a struct field, and I only want metadata.quality_score. However, if metadata is not filled in, the result will return an error.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Python ds.to_batches entry point and reproduce the issue with the columns list shown, including metadata.quality_score without metadata. Trace the struct-column projection handling and verify that selecting the nested field no longer errors when the root field is omitted. No specific source file or test is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100