[BUG] dask_cudf generates files it cannot read back
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Somewhere between `dgdf = dask_cudf.read_csv(..)` and `dgdf.to_parquet()`, the generated files are written in a way that `cudf.read_parquet` and `dask_cudf.read_parquet` will fail to read the data back.
Exception: `cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1607621803079/work/cpp/src/io/parquet/reader_impl.cu:371: All sources must have the same schemas`
**Steps/Code to reproduce bug**
May need to fix paths:
Download:
```bash
curl -O /tmp/logs.csv.gz "https://s3.amazonaws.com/botsdataset/botsv1/csv-by-sourcetype/botsv1.WinEventLog%3ASecurity.csv.gz"
(cd /tmp && gunzip logs.csv.gz)
```
Convert:
```python
with dask.distributed.Client(ADDRESS):
dgdf = dask_cudf.read_csv('/tmp/logs.csv')
dgdf.to_parquet(
'/tmp/logs.parquet',
compression='snappy',
write_index=False,
index=False)
```
Test: Unexpectedly throws exn
```python
cudf.read_parquet('/tmp/logs.parquet')
```
**Expected behavior**
The converted file to read back with matching dtypes... but throws an exn
**Environment overview (please complete the following information)**
RAPIDS 0.18 (conda) in docker (ubuntu); A100's
**Additional context**
* Variants where we set `schema`, `dtypes`, and `use_pandas_metadata` also fail
* Also seeing failures when doing dask_cudf.read_parquet, and doing an intermediate repartition
Contributor guide
Assessment
This issue has not been assessed yet.