lance-format / lance-format/lance
Allow blob API to be used with regular binary columns
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently we require columns to be marked as blobs in order to enable the blob API. This helps us avoid page overhead and makes writing the blobs easier. However, we should still be able to use the blob API (don't read entire blob in memory) even for data types stored as binary. The scheduler can locate the data and then return the description.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the blob API and scheduler paths for columns marked as blobs, then compare how regular binary columns are represented. Done means the blob API can locate and return regular binary values without loading the entire blob into memory, while preserving existing blob-column behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100