[FEA] Profile thread divergence in Parquet Variant `resolve_path` and evaluate warp-collaborative field decode
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
In `cpp/src/io/parquet/experimental/variant_extract.cu`, `locate_variant_fields_kernel` assigns one thread per row, with each thread independently calling `resolve_path` (variant_extract.cu:607) to walk that row's path through the VARIANT metadata/value blobs. However, since the rows can hold differently-shaped Variant data, there is a possibility of warp divergence in this kernel. We are curious as to whether this causes warp divergence with our 100 field microbenchmark (bench_variant_extract_fields, num_fields=100) and whether a warp-collaborative decode strategy would be worth the added complexity.
**Describe the solution you'd like**
Profile `locate_variant_fields_kernel using NSight Compute on the bench_variant_extract_fields, num_fields=100 workload to see whether warp divergence is a real concern. Based on these results, implement the warp-collaborative decode strategy to see whether this is an effective solution to this problem.
**Describe alternatives you've considered**
Leave as-is, if the performance is not affected by this problem.
**Additional context**
Originated from discussion at https://github.com/NVIDIA/cudf/pull/23638#discussion_r3830276215.
Contributor guide
Research direction
Start in cpp/src/io/parquet/experimental/variant_extract.cu, focusing on locate_variant_fields_kernel and resolve_path near line 607. Run Nsight Compute with bench_variant_extract_fields using num_fields=100, then compare the observed divergence and performance with the current behavior. Done means documenting whether divergence is significant and, if justified, evaluating the warp-collaborative decode strategy against the baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100