[FEA] Add logical type inspection for raw Parquet VARIANT values
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Description**
Part of #23182 and the Parquet VARIANT story #22312.
**Is your feature request related to a problem? Please describe.**
`cudf::io::parquet::experimental::cast_variant` returns null when a raw Variant value is null or its encoded type does not match the requested output type. Consumers cannot determine from that result whether the path/value was absent, the value was an encoded Variant null or the value had another logical type.
Downstream engines need a device-side logical type column to build source masks and implement null inspection, source-specific conversion, strict conversion handling, and object/array dispatch without parsing Variant headers themselves.
**Describe the solution you'd like**
Add an experimental libcudf operation next to `cast_variant`:
```cpp
enum class variant_logical_type : uint8_t {
object,
array,
null_value,
boolean,
long_value,
string,
double_value,
decimal,
date,
timestamp,
timestamp_ntz,
float_value,
binary,
uuid
};
[[nodiscard]] std::unique_ptr get_variant_type_id(
column_view const& values,
rmm::cuda_stream_view stream = cudf::get_default_stream(),
rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref());
```
The exact API names, numeric identifiers, and output integral type are open for discussion. We can discuss it in this issue.
The required behavior is:
- Input is a raw Variant `list` column such as the output of `get_variant_field`.
- Output has the same row count and contains stable, engine-neutral logical type identifiers.
- An input-null row produces an output-null row.
- An encoded Variant null produces a valid `null_value` identifier.
- Physical INT1/INT2/INT4/INT8 encodings all produce logical `long_value`.
- Short and long strings both produce logical `string`.
- Object, array, boolean, numeric, temporal, binary, and UUID values produce their corresponding logical identifiers.
- Unknown or malformed input follows a documented error or malformed-value contract and is not silently classified as another valid type.
- Mixed-type, sliced, empty, and all-null columns are covered by tests.
**Additional context**
The implementation can share the Variant header helpers already used by `cast_variant`. The type categories should be based on the Variant format rather than Spark enum ordinals, even though Spark exposes the same logical categories.
Contributor guide
Research direction
Start by reading the existing cast_variant operation and the get_variant_field path, including the shared Variant header helpers mentioned in the issue. Define the logical-type API and malformed-value contract, then cover mixed-type, sliced, empty, and all-null raw Variant columns; done means correct stable identifiers and null handling for every listed Variant category.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100