posit-dev / posit-dev/pointblank
Schema datatype alignment across ibis backends
Open
Nobody has claimed this yet.
Type: ⁇ Question
- Dominant language
- Python
- Stars
- 490
- Forks
- 32
- Avg merge
- 14h 24m
- Merged PRs (30d)
- 2
Description
Prework
- Read and agree to the code of conduct and contributing guidelines.
- If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
Question
What would you like to know?
- Is there a way or could you implement a way to align datatypes across ibis backends?
- I am working on a project that uses ibis to connect to DuckDB while working locally and Spark while working in DataBricks. There are differences in capitalization and differences for array types (
List(String)vsarray<string>) depending on the backend even though the data is the same. Consistency of datatypes across backends would simplify the validation process when using ibis with pointblank.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name files, tests, or entry points. Start by comparing the datatype representations returned through Ibis for DuckDB and Spark, focusing on capitalization and array types. Done should mean a defined, consistent datatype representation that pointblank can use across both backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100