lance-format / lance-format/lance

Support PyArrow sorted MapType schemas

Open
#6,478 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-python enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

Lance currently rejects or loses Arrow MapType schemas with keys_sorted=true in several paths. This blocks round-tripping PyArrow schemas such as:

pa.map_(pa.string(), pa.int32(), keys_sorted=True)

The limitation was originally called out in code comments around map support because Rust/PyArrow schema conversion could drop the keys_sorted property.

Expected behavior

Lance should preserve MapType.keys_sorted for V2.2 map data across write and read paths, including:

  • dataset writes and reads
  • scanner/table/batch reads
  • fragment writes and reads
  • direct file reader APIs
  • Lance schema conversion APIs
  • namespace schema JSON conversion

Notes

The implementation should keep normal unsorted maps unchanged and avoid adding per-batch Python wrapping for schemas that do not need sorted-map restoration.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Rust/PyArrow schema conversion paths and the dataset, scanner/table/batch, fragment, direct file reader, Lance schema, and namespace JSON APIs named in the issue. Verify how V2.2 MapType schemas are handled for sorted and unsorted maps across writes and reads. Done means keys_sorted=true round-trips through every listed path without changing normal maps or adding unnecessary per-batch Python wrapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.