[FEA][Story] Support ingest of Parquet VARIANT columns
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Background
Apache Parquet's [VARIANT logical type](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) is the standard binary encoding for semi-structured data in the Parquet ecosystem. Spark 4.0, DuckDB, and parquet-mr all emit VARIANT-annotated columns when persisting JSON-like data, and Spark-RAPIDS already accepts VARIANT inputs. libcudf today does not recognize the VARIANT logical type at all -- a VARIANT column either fails to parse or is read as an opaque struct of binary children with no way to access its fields on the GPU.
This story tracks adding **GPU-accelerated** support for reading and extracting fields from Parquet VARIANT columns in libcudf and exposing it through pylibcudf, covering both the unshredded layout and Parquet's [shredded VARIANT](https://github.com/apache/parquet-format/blob/master/VariantShredding.md) layout.
A VARIANT column is materialized in libcudf as `struct metadata, list value, ...>`. For unshredded inputs, only the `metadata` and `value` blobs are present. For shredded inputs, additional typed children (`typed_value` subtree) sit alongside the blobs and the per-row value lives in either `value` or one of the typed children depending on whether shredding succeeded for that row. Field extraction is a two-pass GPU kernel: a sizing pass walks the on-device parser to compute output lengths, then an offset scan and copy pass produces the result column. Shredding-aware extraction additionally consults the typed children before falling back to the binary blob.
## Plan
| Status | PR | Scope |
|---|---|---|
| ✅ #22310 | Add Parquet VARIANT reader infrastructure | Recognize the VARIANT logical type (union field 16) in the Thrift compact-protocol reader, route a VARIANT group's `metadata`/`value` BYTE_ARRAY children through the column buffer as `list` via a new internal `PARQUET_COLUMN_BUFFER_FLAG_VARIANT_BINARY`, and guard `sanitize_schema` so it does not rewrite shredded VARIANT children. Reader is shredding-compatible: shredded files load and surface their typed children as ordinary cuDF columns. Pylibcudf shape tests on committed unshredded and shredded VARIANT fixtures. |
| ✅ #22416 | Core GPU variant field extraction (scalar + nested objects) | Public `cudf::io::parquet::experimental::{get_variant_field, cast_variant, extract_variant_field}` APIs that walk a JSONPath-like object-key path (`$?(.name)+`) on-device and decode the leaf value. Supports INT8/INT16/INT32/INT64 and STRING target types. Unshredded only. |
| ✅ #22895 | Array indexing | Extend the `variant_path` parser to accept the full JSONPath subset (`$.foo[2].bar`, `$['weird.key']`) and add name-step / index-step / quoted-key dispatch in the device walk. Inherits the unshredded-only behavior. |
| ⏳ [#TBD](#) | Variant extraction microbenchmarks | NVBench divergence suite with dictionary-scan, field-scan, and found-vs-null scenarios for the kernels delivered in PRs 2-3. |
| ⏳ [#TBD](#) | End-to-end example workload | Standalone `cpp/examples/variant_workload/` exercising the `read_parquet` → `extract_variant_field` flow on representative VARIANT data, demonstrating the public API on realistic inputs. |
| ⏳ [#TBD](#) | Multi-field extraction (issue #22897) | Batched `get_variant_fields` / `extract_variant_fields` entry points returning a `cudf::table` with one column per requested path. Faster than looping `get_variant_field` when paths share prefixes. Unshredded only. |
| ⏳ [#TBD](#) | Shredding-aware variant extraction | Teach `extract_variant_field` (or add a parallel entry point) to honor Parquet's variant shredding fallback rules: per row, prefer the matching `typed_value` child when non-null, otherwise fall back to decoding `value`, otherwise treat as null/missing. |
## Out of scope (future stories)
- **Variant write path** -- producing VARIANT-annotated Parquet output. Today libcudf's parquet writer has no VARIANT case.
- **`extract_variant_field` target types beyond INT{8,16,32,64} / STRING** (FLOAT32/64, BOOL, DECIMAL, TIMESTAMP, nested → typed lists/structs).
- **Variant filter pushdown** -- using a path expression to skip row groups in the parquet reader before materialization.
- **cudf-python and cudf-polars surface** -- this story stops at pylibcudf. A follow-up story tracks `cudf.DataFrame[col].variant.extract(path, dtype)` accessors and a polars `variant.field()` operator.
## References
- Apache Parquet [VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)
- Spark [SPARK-44034 VARIANT data type](https://issues.apache.org/jira/browse/SPARK-44034)
- DuckDB [VARIANT support](https://duckdb.org/docs/data/parquet/overview.html)
- Apache parquet-testing [VARIANT reference vectors](https://github.com/apache/parquet-testing/tree/master/variant)
Contributor guide
Assessment
This issue has not been assessed yet.