[Feature] Support Variant type for PyPaimon
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Paimon Java already supports the VARIANT type, but pypaimon has no equivalent implementation. This PR adds VARIANT read/write support to pypaimon, enabling Python-based compute engines to integrate it. For example, Daft is planning [native VARIANT support](https://docs.daft.ai/en/stable/roadmap/#future-work) at the engine level.
The ideal long-term solution is to wait for PyArrow's official Variant support ([apache/arrow#45937](https://github.com/apache/arrow/issues/45937)), which would guarantee both correctness and performance from the upstream ecosystem. We should continue tracking that progress.
That said, a Python-based implementation now is a reasonable short-term step: users and compute engines can begin integrating VARIANT support early, and the underlying implementation can later be swapped for PyArrow's native one without breaking existing users.
Moreover, some aspects of VARIANT integration are the responsibility of PyPaimon itself and cannot be delegated to PyArrow — for example, shredded column pruning and shredded column predicate pushdown. These will remain part of pypaimon regardless of upstream changes.
### Solution
This work is broken into three parts:
1. **VARIANT read/write support**
2. **Shredded column pruning**
3. **Shredded column predicate pushdown**
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Paimon's existing Java VARIANT support and the corresponding PyPaimon type and read/write paths. Then examine how shredded column pruning and predicate pushdown are handled, while tracking the referenced PyArrow Variant issue. Done means PyPaimon supports VARIANT read/write, shredded column pruning, and shredded column predicate pushdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100