apache / apache/paimon

[Feature] Support Variant type for PyPaimon

Open
#7,655 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Paimon Java already supports the VARIANT type, but pypaimon has no equivalent implementation. This PR adds VARIANT read/write support to pypaimon, enabling Python-based compute engines to integrate it. For example, Daft is planning [native VARIANT support](https://docs.daft.ai/en/stable/roadmap/#future-work) at the engine level.

The ideal long-term solution is to wait for PyArrow's official Variant support ([apache/arrow#45937](https://github.com/apache/arrow/issues/45937)), which would guarantee both correctness and performance from the upstream ecosystem. We should continue tracking that progress.

That said, a Python-based implementation now is a reasonable short-term step: users and compute engines can begin integrating VARIANT support early, and the underlying implementation can later be swapped for PyArrow's native one without breaking existing users.

Moreover, some aspects of VARIANT integration are the responsibility of PyPaimon itself and cannot be delegated to PyArrow — for example, shredded column pruning and shredded column predicate pushdown. These will remain part of pypaimon regardless of upstream changes.

### Solution

This work is broken into three parts:

1. **VARIANT read/write support**
2. **Shredded column pruning**
3. **Shredded column predicate pushdown**

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing Paimon's existing Java VARIANT support and the corresponding PyPaimon type and read/write paths. Then examine how shredded column pruning and predicate pushdown are handled, while tracking the referenced PyArrow Variant issue. Done means PyPaimon supports VARIANT read/write, shredded column pruning, and shredded column predicate pushdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.