lance-format / lance-format/lance
Feature Request: Go SDK for reading Lance datasets
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
Lance currently has Rust, Python, and Java SDKs, but no Go SDK. Go-based data infrastructure projects (filesystems, object storage gateways, data pipelines) that need to interact with Lance datasets have no official way to do so.
Concrete Example: JuiceFS
JuiceFS (10K+ stars) is integrating Lance dataset warmup support. The implementation needs to:
- Parse Lance manifests to discover data files
- Read V2 file footers and CMO tables for column-level metadata
- Map byte ranges to object storage slices for targeted cache warming
Without a Go SDK, we had to manually copy proto files, generate Go code, and write ~800 lines of format parsing logic — duplicating work that already exists in the Rust and Python SDKs.
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing Rust, Python, and Java SDKs and the Lance manifest, V2 file footer, and CMO table handling described in the issue. Define the Go SDK scope around reading Lance datasets and targeted metadata access, with completion demonstrated by supporting the JuiceFS warmup use case without duplicated format-parsing logic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100