huggingface / huggingface/datasets
ArrowInvalid when mapping dataset with missing values
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Describe the bug
I encountered an `ArrowInvalid` when mapping dataset with missing values.
Here are the files for a minimal example. The exception is only thrown when the first line in the csv has a missing value (if you move the last line to the top it isn't thrown).
[data_small.csv](https://github.com/huggingface/datasets/files/7037838/data_small.csv)
[data.csv](https://github.com/huggingface/datasets/files/7037842/data.csv)
## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("csv", data_files=['data_small.csv'])
datasets = datasets.map(lambda e: {'labels': e['match']},
remove_columns=['id'])
```
## Expected results
No error
## Actual results
```
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid null value
```
## Environment info
- `datasets` version: 1.5.0
- Platform: Linux-5.11.0-25-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
Contributor guide
Assessment
This issue has not been assessed yet.