apache / apache/arrow

[Python][Parquet] Support reading and writing Parquet Variant columns

Open
#50,132 1 comment 0 reactions 1 assignee Claimed by @qzyu999 View on GitHub
Component: Parquet Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

### **Description:**

The C++ schema mapping between Parquet's `VARIANT` logical type and Arrow's `VariantExtensionType` (`arrow.parquet.variant`) is established by GH-46104. With the C++ Variant encoder (GH-45947) and decoder (GH-45946) implementations, the underlying C++ Parquet reader/writer will be able to process actual data payloads.

This issue tracks Python-level Parquet integration and testing to ensure Python users can read and write Parquet files containing Variant columns seamlessly.

#### **Proposed Changes:**

1. **Parquet Read/Write Pipeline Validation**:
* Ensure that `pyarrow.parquet.write_table` correctly serializes `VariantExtensionType` columns into Parquet files with the `VARIANT` logical type annotation.
* Ensure that `pyarrow.parquet.read_table` correctly deserializes Parquet `VARIANT` columns back into PyArrow `VariantArray` columns (rather than falling back to the raw binary-pair storage struct or throwing an unsupported type exception).
2. **Metadata and Schema Inspection**:
* Verify that `pyarrow.parquet.read_schema` and `ParquetFile.schema` correctly report the column type as the `VariantExtensionType` extension type.
3. **Integration Testing**:
* Add end-to-end tests in `python/pyarrow/tests/parquet/test_data_types.py` (or a dedicated integration test suite):
* Construct a `pyarrow.Table` containing a `VariantArray` (e.g., from nested dictionaries/lists).
* Write it to a file using `pyarrow.parquet.write_table`.
* Read the file back using `pyarrow.parquet.read_table` and assert that the types and values are identical to the original table.
* Test reading a reference Parquet file containing Variant data written by a different implementation (e.g., Go or Spark) to verify cross-language compatibility.

#### **Dependencies:**

This issue is blocked by:
* **GH-50131**: [Python] Bindings for Variant canonical extension type (Exposing the Python `VariantType`, `VariantArray`, and `VariantScalar` classes)
* **GH-45946** (PR #50121): [C++][Parquet] Variant decoding
* **GH-45947** (PR #50122): [C++][Parquet] Variant encoding

### Component(s)

Python, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.