lance-format / lance-format/lance
Provide adaptor to coerce input schema for append
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Sometimes it's challenging to produce the exact same schema for appends. For example, we require dictionary columns to have exactly the same dictionary, but other Arrow systems expect the dictionary to be flexible batch-to-batch. Other examples might be timestamp resolutions, where Python users might not know what resolution their particular system is producing.
We could provide a RecordBatchReader that wraps the input and attempts to coerce data to the correct schema.
We might want to make this default in the Python API.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The proposal names a RecordBatchReader adaptor around append inputs and a possible Python API default, but no files or tests. Start by locating the append path and Python API, then determine how schema coercion should handle dictionary columns and timestamp resolutions. Done should include agreed coercion behavior and coverage for these examples, including whether the adaptor is the default in Python.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100