NVIDIA / NVIDIA/cudf

Support logical type inspection for Parquet VARIANT values

Open
#23,182 0 comments 0 reactions 0 assignees View on GitHub
cuIO feature request Java libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

### Background
Part of the Parquet VARIANT work tracked by #22312.

The experimental Parquet VARIANT APIs can extract raw `list` value blobs and decode exact STRING and INT8/INT16/INT32/INT64 values. They do not currently expose the logical type of each raw Variant value.

An exact decoder returns null when the input row is null or its encoded type does not match the requested type. A consumer therefore cannot use decoder validity alone to distinguish an absent value, an encoded Variant null and a present value of another logical type.

Logical type inspection is needed by downstream engines to implement Variant-null checks, build source-type masks for exact decoding and conversion, distinguish benign null/missing results from strict conversion failures and dispatch object/array/scalar processing without duplicating Variant header parsing outside cuDF.

This issue tracks the libcudf operation and its public cuDF Java exposure.

- [ ] Add an experimental libcudf operation over raw Variant `list` values that returns a device column of logical type identifiers. Include null/malformed behavior, C++ tests and documentation - https://github.com/rapidsai/cudf/issues/23183
- [ ] Expose the libcudf operation through `ai.rapids.cudf.VariantUtils` and the cuDF-owned JNI bridge. Include public type constants, Java tests and documentation - https://github.com/rapidsai/cudf/issues/23184
- [ ] Add libcudf structured outcome status for VARIANT extraction and conversion - https://github.com/rapidsai/cudf/issues/23467
- [ ] Expose VARIANT outcome status through `ai.rapids.cudf.VariantUtils`

## References

- Parquet Variant encoding: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
- Existing Variant extraction APIs: #22416
- Existing cuDF Java Variant APIs: #23069
- Spark logical Variant types: https://github.com/apache/spark/blob/v4.1.2/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java#L237-L252

Contributor guide

Open the contributing guide

Research direction

This umbrella issue covers a libcudf operation and its Java exposure through ai.rapids.cudf.VariantUtils; start by reading the existing Variant APIs in #22416 and #23069, then review the scoped work in #23183 and #23184. Done means logical type identifiers, defined null and malformed behavior, C++ and Java tests, public constants, and documentation across both layers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
api, backend-api-design, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.