NVIDIA / NVIDIA/cudf

[FEA] Add opt-in per-row status for VARIANT extraction and checked conversion

Open
#23,467 0 comments 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
The current VARIANT extraction and exact-decoding APIs use a null output for several different cases: SQL-null input, missing path, encoded VARIANT null, and source-type mismatch. Checked conversions can add other causes, including overflow and invalid conversion.

Logical-type inspection from #23183 tells callers what a raw VARIANT value contains, but not what happened during a particular extraction or conversion. Once extraction returns a null raw value, callers cannot recover the missing-path outcome from the extracted column alone.

Downstream engines need these outcomes to remain distinguishable on the device without reparsing the VARIANT value or inferring the cause from output validity.

**Describe the solution you'd like**
Add an opt-in status output to the existing VARIANT extraction and exact-decoding APIs, and to checked-conversion APIs as they are added. Existing value-only calls should continue to return only the value column and retain their current behavior.

The status column should only be materialized when requested, so existing value-only callers do not incur the additional output column allocation. This could be an optional output parameter, a separate overload, or an equivalent mechanism. The exact API shape is open for discussion, but callers should not have to allocate the status column unless they request it.

When requested, the value and status columns must have the same row count and row order. The status values should be equivalent to:

```
enum class variant_operation_status : uint8_t {
success,
missing_path,
variant_null,
type_mismatch,
malformed_variant,
overflow,
invalid_conversion
};
```
A SQL-null original input with no incoming extraction status produces a null value and null status. Every other row receives one valid status:

- `success`: the requested output was produced.

- `missing_path`: path resolution did not complete because a key was absent, an array index was out of range, or a non-container/VARIANT-null value was encountered before the final path step.

- `variant_null`: the complete path resolved to an encoded VARIANT null or a direct raw input was an encoded VARIANT null.

- `type_mismatch`: the requested operation and target are supported, but this row’s resolved source type is not accepted.

- `malformed_variant`: bytes or metadata needed by the requested operation are invalid or truncated. Bytes unrelated to the requested value do not need to be validated.

- `overflow`: a supported conversion is outside the target range or precision.

- `invalid_conversion`: a supported conversion failed for another value-dependent reason, such as non-numeric content for a numeric target.

For typed results, only success produces a valid value. All other outcomes produce a null result. Raw extraction should preserve an encoded VARIANT-null value and return variant_null instead of converting it to SQL null.

cuDF should reject invalid native requests, such as an unsupported target type, invalid path representation, or incorrect input column layout. When status is requested, data-dependent problems should be reported per row without failing otherwise valid rows. Calls that do not request status should retain the existing value-only behavior.

When status is requested and carried into later decoding or conversion, a null incoming status must remain null, and any existing non-success status must be preserved. Only rows whose upstream status is `success` should be decoded or converted. This can be implemented through a combined API or by passing the status with the extracted values.

This API does not replace get_variant_type_id from #23183 and does not need to return the source logical type. The implementation should reuse the same VARIANT parsing helpers where possible.

The opt-in status path should also be exposed through the public `ai.rapids.cudf.VariantUtils` API and the cuDF-owned JNI bridge so Java consumers such as cudf-spark can use it. Existing Java value-only methods should remain unchanged. The status identifiers should be public, documented, and stable, with matching Java constants or a small status definition. The exact Java result shape is open for discussion.

### cuDF-Spark usage

The status lets cudf-spark distinguish missing values, VARIANT nulls, and conversion failures directly on the GPU, without extra parsing.

cuDF should provide engine-neutral decoding and conversion outcomes. Spark-specific conversion rules, strict-versus-tolerant behavior, and error handling remain in cudf-spark.

cuDF-Spark will not request status for every operation. Nullable results are generally sufficient for tolerant operations such as try_variant_get, while strict conversions need the additional status to distinguish conversion failures from missing paths and VARIANT nulls.

### Testing
- Tests distinguish SQL null, terminal VARIANT null, null before path completion, missing paths, type mismatch, malformed payloads, overflow, and invalid conversion.

- Mixed, all-null, empty, and sliced inputs are covered and a row-level data failure does not abort unrelated rows.

- Both API paths are covered: calls without status retain the existing result and behavior, while opt-in calls return a correctly aligned status column.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing VARIANT extraction and exact-decoding APIs and their shared parsing helpers, then inspect the public ai.rapids.cudf.VariantUtils API and cuDF-owned JNI bridge. Define the opt-in result path while preserving existing value-only behavior. Add coverage for the listed row-level outcomes, null and sliced inputs, aligned status columns, and unchanged calls without status.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
api, backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.