NVIDIA / NVIDIA/cudf

[BUG] dask_cudf.read_json seems to be freezing when given a path with large number of files

Open
#12,951 8 comments 0 reactions 0 assignees View on GitHub
bug cuIO dask libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

dask_cudf.read_json seems to be freezing when given a path with large number of files. Providing the list of files directly works

Below seems to be freezing
```python3
text_ddf = dask_cudf.read_json(f'{INPUT_PATH}/data/*',engine='cudf',lines=True)
```

Below works
```python3
files = list(map(lambda x: os.path.join(data_path, x), os.listdir(data_path)))

text_ddf = dask_cudf.read_json(files,engine='cudf',lines=True)
```

**Additional context**
CC: @ayushdg

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.