Deeply nested null-only column creation can be slow with factory functions
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In Spark-RAPIDS, when there are input files of different schemas, the file reader can create a merged schema and fill the missing columns with nulls in the output batch. To do so, it currently creates the column view first for the missing columns, and converts it to the column by calling `ColumnView.copyToColumnVector()`, which is eventually just simply `new cudf::column(view)`. It appears that adding missing columns with nulls can take a significant time when the schema has deeply nested columns. For example, when the column type is the struct, it needs to call `superimpose_and_sanitize_nulls` for its all descendants if it has a null mask. The screenshot below is an nsys trace of a Spark task with the plugin. The parquet decode block is the range that read a batch from parquet files. The following many tiny blocks are the ranges to add missing columns. As seen in the screenshot, adding missing columns was as big as the parquet decode itself. But I have seen that this post processing took even longer than the parquet decode. The small blocks are mostly `superimpose_and_sanitize_nulls` and `superimpose_nulls`. The merged schema was deeply nested. It had 8 top-level fields but 9430 fields total. Among them, 1179 were array columns and 1817 were struct columns. The max level of the nested field was 11.
Contributor guide
Assessment
This issue has not been assessed yet.