NVIDIA / NVIDIA/cudf

[BUG] Incorrect read_parquet on spark distributed parquet files.

Open
#16,968 2 comments 0 reactions 0 assignees View on GitHub
bug cuIO dask Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When I read in a parquet dataset saved with Spark on a databricks catalog I get lots of .
I tried

```
import glob

cudf_dfs = [cudf.read_parquet(file) for file in glob.glob("/Volumes/path/*.parquet")]
cudf_df = cudf.concat(cudf_dfs,ignore_index=True)
```

and

```
import dask_cudf
dask_df = dask_cudf.read_parquet("/Volumes/path/*.parquet",chunksize='50MB')
cudf_df = dask_df.compute()

```

and I also tried using the pyarrow engine on the load which overflows.

Let me know if you need some code to replicate but it should be easy. Any spark based parquet would do.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.