NVIDIA / NVIDIA/cudf

Deeply nested null-only column processing can be slow in cudf::concatenate

Open
#22,774 2 comments 0 reactions 0 assignees View on GitHub
improvement libcudf Performance Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In Spark-RAPIDS, the merge sort (`SortUtils.mergeSortAndCloseWithRetry()`) has a fallback route that it concatenates all batches first and then sort the concatenated batch. This can happen when the key column includes nested type columns such as struct, map, array, or binary. This fallback can take a significant time when the schema has deeply nested columns. The below screenshot shows that the concatenate took 24 seconds out of 25 seconds of the merge sort. The schema was deeply nested. It had 8 top-level fields but 9430 fields total. Among them, 1179 were array columns and 1817 were struct columns. The max level of the nested field was 11.

Image

The below is a table of the operations under the concatenate block in the screenshot.

| count | total s | max s | type |
|---:|---:|---:|---|
| 592,658 | 3.000745619 | 0.072135998 | `cudaStreamSynchronize_v3020` |
| 428,596 | 2.078729524 | 0.033403695 | `cudaMemcpyAsync_ptsz_v7000` |
| 200,813 | 1.336713403 | 0.001239211 | `cudaLaunchKernel_ptsz_v7000` |
| 184,396 | 0.177945291 | 0.000080811 | `cudaEventRecord_ptsz_v7000` |
| 105,963 | 0.204307859 | 0.007813688 | `mutable_view` |
| 16,043 | 0.314488050 | 0.085258360 | `cudaStreamSynchronize_ptsz_v7000` |
| 15,392 | 6.131321785 | 0.353934142 | `superimpose_nulls` |
| 9,954 | 0.008119848 | 0.000390815 | `cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags_v7000` |
| 5,256 | 0.044805619 | 0.000480961 | `empty_like` |
| 4,840 | 0.117998327 | 0.001234814 | `cub::DeviceReduce::Reduce` |
| 3,892 | 0.593389243 | 0.093105082 | `purging` |
| 3,892 | 2.915194098 | 0.432958337 | `superimpose_and_sanitize_nulls` |
| 3,479 | 0.146676266 | 0.001692749 | `allocate_like` |
| 3,002 | 0.002597303 | 0.000031243 | `cudaMemsetAsync_ptsz_v7000` |
| 1,239 | 0.032970994 | 0.000938537 | `make_numeric_column` |
| 966 | 0.028399326 | 0.000939300 | `make_fixed_width_column` |
| 848 | 0.011316946 | 0.000207997 | `create_null_mask` |
| 273 | 0.103686757 | 0.009164404 | `concatenate` |
| 273 | 0.057129871 | 0.008916806 | `create_strings_device_views` |
| 273 | 0.000152608 | 0.000004568 | `make_strings_column` |
| 49 | 0.035334600 | 0.001202934 | `Unknown` |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.