huggingface / huggingface/datasets
Error in saving the PIL image into *.arrow files using datasets.arrow_writer
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I am using the ArrowWriter from datasets.arrow_writer to save a json-style file as arrow files. Within the dictionary, it contains a feature called "image" which is a list of PIL.Image objects.
I am saving the json using the following script:
```
def save_to_arrow(path,temp):
with ArrowWriter(path=path,writer_batch_size=20) as writer:
writer.write_batch(temp)
writer.finalize()
```
However, when I attempt to restore the dataset and use the ```Dataset.from_file(path)``` function to load the arrow file, there seems to be an issue with the PIL.Image object in the dataset. The list of PIL.Images appears as follows rather than a normal PIL.Image object:

### Steps to reproduce the bug
1. Storing the data json into arrow files:
```
def save_to_arrow(path,temp):
with ArrowWriter(path=path,writer_batch_size=20) as writer:
writer.write_batch(temp)
writer.finalize()
save_to_arrow( path, json_file )
```
2. try to load the arrow file into the Dataset object using the ```Dataset.from_file(path)```
### Expected behavior
Except to saving the contained "image" feature as a list PIL.Image objects as the arrow file. And I can restore the dataset from the file.
### Environment info
- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.4.4
Contributor guide
Assessment
This issue has not been assessed yet.