NVIDIA / NVIDIA/cudf

[BUG] dask_cudf generates files it cannot read back

Open
#7,602 9 comments 1 reaction 0 assignees View on GitHub
bug cuIO libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Somewhere between `dgdf = dask_cudf.read_csv(..)` and `dgdf.to_parquet()`, the generated files are written in a way that `cudf.read_parquet` and `dask_cudf.read_parquet` will fail to read the data back.

Exception: `cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1607621803079/work/cpp/src/io/parquet/reader_impl.cu:371: All sources must have the same schemas`

**Steps/Code to reproduce bug**

May need to fix paths:

Download:

```bash
curl -O /tmp/logs.csv.gz "https://s3.amazonaws.com/botsdataset/botsv1/csv-by-sourcetype/botsv1.WinEventLog%3ASecurity.csv.gz"
(cd /tmp && gunzip logs.csv.gz)
```

Convert:
```python
with dask.distributed.Client(ADDRESS):
dgdf = dask_cudf.read_csv('/tmp/logs.csv')
dgdf.to_parquet(
'/tmp/logs.parquet',
compression='snappy',
write_index=False,
index=False)
```

Test: Unexpectedly throws exn
```python
cudf.read_parquet('/tmp/logs.parquet')
```

**Expected behavior**
The converted file to read back with matching dtypes... but throws an exn

**Environment overview (please complete the following information)**
RAPIDS 0.18 (conda) in docker (ubuntu); A100's

**Additional context**

* Variants where we set `schema`, `dtypes`, and `use_pandas_metadata` also fail
* Also seeing failures when doing dask_cudf.read_parquet, and doing an intermediate repartition

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.