[FEA][Java][JNI] Expose Parquet VARIANT DECIMAL decoding
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
After libcudf adds Parquet VARIANT DECIMAL decoding - https://github.com/NVIDIA/cudf/issues/23817, cudf-spark still needs a public cuDF Java/JNI path to request it. The current JNI methods pass only a native type ID; JNI cannot construct a decimal `cudf::data_type` without its scale.
Without this binding, `try_variant_get(v, '$.amount', 'decimal(18,2)')` must remain on CPU even though libcudf can decode the payload.
**Describe the solution you'd like**
Extend `ai.rapids.cudf.VariantUtils` and its cuDF-owned JNI bridge to support the finalized libcudf DECIMAL API. Target-decimal calls should retain the existing public shape:
```java
DType decimal = DType.create(DType.DTypeEnum.DECIMAL64, -2);
try (ColumnVector value =
VariantUtils.castVariantValue(valueBytes, decimal);
ColumnVector extracted =
VariantUtils.extractVariantField(variantStruct, path, decimal)) {
// consume value and extracted
}
```
Pass both type ID and scale through JNI and construct the complete native `cudf::data_type`. A cuDF scale of `-2` represents two fractional digits. Spark declared precision remains a cudf-spark responsibility.
If the native issue exposes a lossless mixed-scale source representation for DECIMAL-to-other-target conversion, add the corresponding owning Java result—for example, a table of unscaled values and per-row scale—rather than discarding that information in JNI.
Follow the existing `VariantUtils` conventions for argument validation, device selection, default stream/current memory resource, exception translation, ownership and caller-managed `ColumnVector.close()`.
**Describe alternatives you've considered**
**Additional context**
Definition of done:
- Java and JNI tests cover Variant source scales through each encoding's maximum precision: DECIMAL4 0–9, DECIMAL8 0–18, and DECIMAL16 0–38.
- Tests cover translation to the corresponding cuDF scale, mixed per-row source scales, width boundaries, rescaling/rounding, overflow, mixed types, null/type mismatch, slices, empty/all-null inputs, and invalid arguments.
- Any lossless source-decoding result preserves row alignment, unscaled values, scale, ownership and cleanup semantics.
- Java results and exceptions match the native API.
Contributor guide
Research direction
Start with ai.rapids.cudf.VariantUtils and its cuDF-owned JNI bridge, then confirm the finalized libcudf DECIMAL API from issue 23817. Trace existing validation, stream, memory-resource, exception, ownership, and ColumnVector conventions. Done means Java and JNI tests cover the listed scales, conversions, mixed types, errors, slices, empty and null inputs, while any lossless result preserves row alignment and cleanup semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100