Support logical type inspection for Parquet VARIANT values
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
### Background
Part of the Parquet VARIANT work tracked by #22312.
The experimental Parquet VARIANT APIs can extract raw `list` value blobs and decode exact STRING and INT8/INT16/INT32/INT64 values. They do not currently expose the logical type of each raw Variant value.
An exact decoder returns null when the input row is null or its encoded type does not match the requested type. A consumer therefore cannot use decoder validity alone to distinguish an absent value, an encoded Variant null and a present value of another logical type.
Logical type inspection is needed by downstream engines to implement Variant-null checks, build source-type masks for exact decoding and conversion, distinguish benign null/missing results from strict conversion failures and dispatch object/array/scalar processing without duplicating Variant header parsing outside cuDF.
This issue tracks the libcudf operation and its public cuDF Java exposure.
- [ ] Add an experimental libcudf operation over raw Variant `list` values that returns a device column of logical type identifiers. Include null/malformed behavior, C++ tests and documentation - https://github.com/rapidsai/cudf/issues/23183
- [ ] Expose the libcudf operation through `ai.rapids.cudf.VariantUtils` and the cuDF-owned JNI bridge. Include public type constants, Java tests and documentation - https://github.com/rapidsai/cudf/issues/23184
- [ ] Add libcudf structured outcome status for VARIANT extraction and conversion - https://github.com/rapidsai/cudf/issues/23467
- [ ] Expose VARIANT outcome status through `ai.rapids.cudf.VariantUtils`
## References
- Parquet Variant encoding: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
- Existing Variant extraction APIs: #22416
- Existing cuDF Java Variant APIs: #23069
- Spark logical Variant types: https://github.com/apache/spark/blob/v4.1.2/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java#L237-L252
Contributor guide
Research direction
This umbrella issue covers a libcudf operation and its Java exposure through ai.rapids.cudf.VariantUtils; start by reading the existing Variant APIs in #22416 and #23069, then review the scoped work in #23183 and #23184. Done means logical type identifiers, defined null and malformed behavior, C++ and Java tests, public constants, and documentation across both layers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- api, backend-api-design, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100