NVIDIA / NVIDIA/cudf

[FEA][Story] Convert JSON reader output to Apache VARIANT representation

Open
#23,251 0 comments 0 reactions 1 assignee Claimed by @abigalekim View on GitHub
feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

# [FEA][Story] Convert JSON representations (table/strings) to Parquet VARIANT representation

## Background

Apache's VARIANT logical type provides a binary representation for semi-structured data. The work to ingest these columns and expose their materialized type as a struct of two uint8 byte lists (shown below) is already being done at https://github.com/rapidsai/cudf/issues/22312. cuDF Spark is also working on supporting the Variant type in Spark RAPIDS (https://github.com/NVIDIA/cudf-spark/issues/15171).
```text
struct<
metadata: list,
value: list,
...
>
```

The JSON reader (`read_json`) currently parses JSON inputs into a table of the set of columns along with relevant metadata. For record style inputs, top-level JSON object fields become columns with primitive types (e.g., int, bool, string, float), JSON arrays are represented as list columns, and nested JSON objects are represented as struct columns. This story tracks adding a GPU-accelerated conversion path from the table produced by read_json into the Apache VARIANT representation. This allows users who previously have their data stored as serialized JSON object strings to start to use Variant functionality.

## Plan

| Status | PR | Scope |
|---|---|---|
| ⏳ #23400, #23614 | Core infrastructure for scalar, non-nested JSON to variant converter | Public `cudf::io::parquet::experimental::{encode_variant, encode_strings_to_variant}` API that takes in a table produced by JSON reader (`cudf::io::read_json`) and converts it to a variant column. Supports only scalar, non-nested types (NULLS, integers, floating-point, boolean, strings). Only supports JSON data < 2GB. `encode_strings_to_variant` should take in a string column of JSON strings and use the same underlying logic.|
| ⏳ [#TBD](#) | List support | Extend the converter to handle JSON arrays represented as cuDF list columns. Encode each list row as a Variant array, including scalar lists, null lists, empty lists, and null elements within lists. Support nested list children only where the child type is already supported by the converter. |
| ⏳ [#TBD](#) | Nested structs | Extend the converter to handle JSON objects represented as cuDF struct columns. Walk the nested table/column hierarchy using the JSON reader metadata for field names and recursively encode struct children as Variant object fields. |
| ⏳ [#TBD](#) | Large data support | Extend the converter to support Variant payloads larger than the 2GB column-size limit by adding more `value` columns when a single `value` column cannot contain all encoded bytes. |
| ⏳ [#TBD](#) | Variant converter microbenchmarks | NVBench divergence suite with scalar encoding, nested object encoding, array heavy inputs for the kernels in PRs 1-4. |
| ⏳ [#TBD](#) | End-to-end example workload | Standalone `cpp/examples/variant_workload/` exercising the `read_json` → `encode_variant` flow on representative VARIANT data, demonstrating the public API on realistic inputs. |
| ⏳ [#TBD](#) | Shredding-aware variant conversion | Add support for shredded columns in the converter, where selected fields are materialized as typed columns alongside the unshredded `metadata` and `value` representation. Add public `cudf::io::parquet::experimental::encode_variant_shredded` that takes in a table produced by JSON reader and a list of field names to be converted into shredded columns.|

## Out of Scope (future stories)
- Variant writer: Producing Variant-annotated Parquet output. The logic used for conversion into Variant and the encoding should be reusable when we implement a writer.

## References
- [`read_json` API](https://docs.rapids.ai/api/libcudf/legacy/group__io__readers.html#ga718a34e77834c130ede642e152358832)
- Apache Parquet [VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [VariantShredding.md](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.