[Python][Parquet] Support reading and writing Parquet Variant columns
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
### **Description:**
The C++ schema mapping between Parquet's `VARIANT` logical type and Arrow's `VariantExtensionType` (`arrow.parquet.variant`) is established by GH-46104. With the C++ Variant encoder (GH-45947) and decoder (GH-45946) implementations, the underlying C++ Parquet reader/writer will be able to process actual data payloads.
This issue tracks Python-level Parquet integration and testing to ensure Python users can read and write Parquet files containing Variant columns seamlessly.
#### **Proposed Changes:**
1. **Parquet Read/Write Pipeline Validation**:
* Ensure that `pyarrow.parquet.write_table` correctly serializes `VariantExtensionType` columns into Parquet files with the `VARIANT` logical type annotation.
* Ensure that `pyarrow.parquet.read_table` correctly deserializes Parquet `VARIANT` columns back into PyArrow `VariantArray` columns (rather than falling back to the raw binary-pair storage struct or throwing an unsupported type exception).
2. **Metadata and Schema Inspection**:
* Verify that `pyarrow.parquet.read_schema` and `ParquetFile.schema` correctly report the column type as the `VariantExtensionType` extension type.
3. **Integration Testing**:
* Add end-to-end tests in `python/pyarrow/tests/parquet/test_data_types.py` (or a dedicated integration test suite):
* Construct a `pyarrow.Table` containing a `VariantArray` (e.g., from nested dictionaries/lists).
* Write it to a file using `pyarrow.parquet.write_table`.
* Read the file back using `pyarrow.parquet.read_table` and assert that the types and values are identical to the original table.
* Test reading a reference Parquet file containing Variant data written by a different implementation (e.g., Go or Spark) to verify cross-language compatibility.
#### **Dependencies:**
This issue is blocked by:
* **GH-50131**: [Python] Bindings for Variant canonical extension type (Exposing the Python `VariantType`, `VariantArray`, and `VariantScalar` classes)
* **GH-45946** (PR #50121): [C++][Parquet] Variant decoding
* **GH-45947** (PR #50122): [C++][Parquet] Variant encoding
### Component(s)
Python, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.