Deeply nested null-only column processing can be slow in cudf::concatenate
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In Spark-RAPIDS, the merge sort (`SortUtils.mergeSortAndCloseWithRetry()`) has a fallback route that it concatenates all batches first and then sort the concatenated batch. This can happen when the key column includes nested type columns such as struct, map, array, or binary. This fallback can take a significant time when the schema has deeply nested columns. The below screenshot shows that the concatenate took 24 seconds out of 25 seconds of the merge sort. The schema was deeply nested. It had 8 top-level fields but 9430 fields total. Among them, 1179 were array columns and 1817 were struct columns. The max level of the nested field was 11.
The below is a table of the operations under the concatenate block in the screenshot.
| count | total s | max s | type |
|---:|---:|---:|---|
| 592,658 | 3.000745619 | 0.072135998 | `cudaStreamSynchronize_v3020` |
| 428,596 | 2.078729524 | 0.033403695 | `cudaMemcpyAsync_ptsz_v7000` |
| 200,813 | 1.336713403 | 0.001239211 | `cudaLaunchKernel_ptsz_v7000` |
| 184,396 | 0.177945291 | 0.000080811 | `cudaEventRecord_ptsz_v7000` |
| 105,963 | 0.204307859 | 0.007813688 | `mutable_view` |
| 16,043 | 0.314488050 | 0.085258360 | `cudaStreamSynchronize_ptsz_v7000` |
| 15,392 | 6.131321785 | 0.353934142 | `superimpose_nulls` |
| 9,954 | 0.008119848 | 0.000390815 | `cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags_v7000` |
| 5,256 | 0.044805619 | 0.000480961 | `empty_like` |
| 4,840 | 0.117998327 | 0.001234814 | `cub::DeviceReduce::Reduce` |
| 3,892 | 0.593389243 | 0.093105082 | `purging` |
| 3,892 | 2.915194098 | 0.432958337 | `superimpose_and_sanitize_nulls` |
| 3,479 | 0.146676266 | 0.001692749 | `allocate_like` |
| 3,002 | 0.002597303 | 0.000031243 | `cudaMemsetAsync_ptsz_v7000` |
| 1,239 | 0.032970994 | 0.000938537 | `make_numeric_column` |
| 966 | 0.028399326 | 0.000939300 | `make_fixed_width_column` |
| 848 | 0.011316946 | 0.000207997 | `create_null_mask` |
| 273 | 0.103686757 | 0.009164404 | `concatenate` |
| 273 | 0.057129871 | 0.008916806 | `create_strings_device_views` |
| 273 | 0.000152608 | 0.000004568 | `make_strings_column` |
| 49 | 0.035334600 | 0.001202934 | `Unknown` |
Contributor guide
Assessment
This issue has not been assessed yet.