dask / dask/dask

dask read_csv ignore broken files

Open
#9,529 4 comments 0 reactions 0 assignees View on GitHub
dataframe feature io needs attention
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

I try to read the CSV files with,
```python
import dask.dataframe as dd
data = dd.read_csv("*.csv.gz",sep='\t',dtype='unicode', low_memory='true',compression='gzip',blocksize=None)
```
I have some CSV files which are broken files. I want to ignore that. It will be great if we have some function like below,

```python
import dask.dataframe as dd
data = dd.read_csv("*.csv.gz",sep='\t',dtype='unicode', low_memory='true',compression='gzip',blocksize=None,ignore_broken_files = True)
```
Which will skip the files that can't be read.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.