NVIDIA / NVIDIA/cudf

[FEA] Support nested struct columns in ORC fuzz tests

Open
#11,483 1 comment 1 reaction 0 assignees View on GitHub
cuIO feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Fuzz testing support for nested struct columns in Orc is incomplete. Some patches were required as shown in #9395 discussion (https://github.com/rapidsai/cudf/issues/9395#issuecomment-1206776202.)

**Describe the solution you'd like**
Add support for nested struct columns in the Orc fuzz testing

**Describe alternatives you've considered**
Adhoc patches that bring side-effects to other fuzzers and unit tests in cuDF.

**Additional context**
We would benefit from a way to limit the overall depth of the generated struct columns - I propose maximum nesting depth of 256. Setting `max_structs_nesting_depth` in the `IOFuzz` class does not seem sufficient to address this limit. Here is a traceback for the recursion error:
```
File "/home/gregorykimball/Repo/cudf/python/cudf/cudf/testing/dataset_generator.py", line 387, in rand_dataframe
structDtype = create_nested_struct_type(
...
File "/home/gregorykimball/Repo/rapids-compose/etc/conda/cuda_11.5/envs/rapids/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 70, in _wrapreduction
passkwargs = {k: v for k, v in kwargs.items()
RecursionError: maximum recursion depth exceeded while calling a Python object

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.