datafusion-contrib / datafusion-contrib/orc-rust
Support decoding Union column with up to including 256 variants
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 30
- Forks
- 26
- Avg merge
- 19m
- Merged PRs (30d)
- 3
Description
According to https://orc.apache.org/specification/ORCv1/
Currently ORC union types are limited to 256 variants, which matches the Hive type model.
However in Arrow, UnionArrays are limited to 127 variants: https://arrow.apache.org/docs/format/Columnar.html#union-layout
A union with more than 127 possible types can be modeled as a union of unions.
To support this, would need to do as above and decode into union of union
See initial Union support here: https://github.com/datafusion-contrib/datafusion-orc/commit/ee69b91cb2ce4c18ee5148bd541658d362fc577f
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the initial Union support commit linked in the issue, then read the ORC union specification and Arrow union layout documentation. The work is done when ORC unions with up to 256 variants can be decoded by representing larger unions as nested unions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100