apache / apache/iceberg-python
Add `VariantType`
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
The VariantType is a semi-structured type that's part of V3.
- Hopefully, the Parquet format will be released soon: https://lists.apache.org/thread/mr8krzdy8qslhhy5zbll53fd2hy6cpwg
- Once that's in, I'll be added to Arrow: https://github.com/apache/arrow/pull/45375
- I think for PyIceberg, we need to wait for PyArrow support.
There is a Python implementation in Spark that we could vendor into the project, but that will be pretty slow: https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py
Thoughts?
Original proposal: https://github.com/apache/iceberg/issues/10392
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by checking the linked Parquet and Arrow work to determine whether the required VariantType support is available, then read Spark's python/pyspark/sql/variant_utils.py as the referenced implementation. Done means PyIceberg has an agreed VariantType implementation path once the PyArrow dependency supports it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100