lance-format / lance-format/lance
Add schema support for pyarrow map / struct / dictionary type
Open
Nobody has claimed this yet.
arrow
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
lancedb API throws TypeError: Converting Pydantic type to Arrow Type: unsupported type typing.Dict[str, str]. ( same for Dict[int,str] ) . Issue is in lance not supporting the pyarrow map types.
To Reproduce :
from lancedb.pydantic import Vector, LanceModel, Dict
class Content(LanceModel):
id: int
vector: Vector(1)
metadata: Dict[int, str]
db = lancedb.connect("./lancedb")
tbl = db.create_table("test", schema=Content.to_arrow_schema())
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example through LanceModel.to_arrow_schema() with Dict[str, str] and Dict[int, str]. Trace the schema conversion path for the unsupported PyArrow map, struct, and dictionary types; done means these declarations produce a valid Arrow schema and the create_table example succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100