NVIDIA / NVIDIA/cudf

[FEA] Profile thread divergence in Parquet Variant `resolve_path` and evaluate warp-collaborative field decode

Open
#23,785 0 comments 0 reactions 0 assignees View on GitHub
? - Needs Triage feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
In `cpp/src/io/parquet/experimental/variant_extract.cu`, `locate_variant_fields_kernel` assigns one thread per row, with each thread independently calling `resolve_path` (variant_extract.cu:607) to walk that row's path through the VARIANT metadata/value blobs. However, since the rows can hold differently-shaped Variant data, there is a possibility of warp divergence in this kernel. We are curious as to whether this causes warp divergence with our 100 field microbenchmark (bench_variant_extract_fields, num_fields=100) and whether a warp-collaborative decode strategy would be worth the added complexity.

**Describe the solution you'd like**
Profile `locate_variant_fields_kernel using NSight Compute on the bench_variant_extract_fields, num_fields=100 workload to see whether warp divergence is a real concern. Based on these results, implement the warp-collaborative decode strategy to see whether this is an effective solution to this problem.

**Describe alternatives you've considered**
Leave as-is, if the performance is not affected by this problem.

**Additional context**
Originated from discussion at https://github.com/NVIDIA/cudf/pull/23638#discussion_r3830276215.

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/io/parquet/experimental/variant_extract.cu, focusing on locate_variant_fields_kernel and resolve_path near line 607. Run Nsight Compute with bench_variant_extract_fields using num_fields=100, then compare the observed divergence and performance with the current behavior. Done means documenting whether divergence is significant and, if justified, evaluating the warp-collaborative decode strategy against the baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.