lance-format / lance-format/lance
Support PyArrow sorted MapType schemas
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
Lance currently rejects or loses Arrow MapType schemas with keys_sorted=true in several paths. This blocks round-tripping PyArrow schemas such as:
pa.map_(pa.string(), pa.int32(), keys_sorted=True)
The limitation was originally called out in code comments around map support because Rust/PyArrow schema conversion could drop the keys_sorted property.
Expected behavior
Lance should preserve MapType.keys_sorted for V2.2 map data across write and read paths, including:
- dataset writes and reads
- scanner/table/batch reads
- fragment writes and reads
- direct file reader APIs
- Lance schema conversion APIs
- namespace schema JSON conversion
Notes
The implementation should keep normal unsorted maps unchanged and avoid adding per-batch Python wrapping for schemas that do not need sorted-map restoration.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Rust/PyArrow schema conversion paths and the dataset, scanner/table/batch, fragment, direct file reader, Lance schema, and namespace JSON APIs named in the issue. Verify how V2.2 MapType schemas are handled for sorted and unsorted maps across writes and reads. Done means keys_sorted=true round-trips through every listed path without changing normal maps or adding unnecessary per-batch Python wrapping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100