huggingface / huggingface/datasets

Loading CSV exported dataset has unexpected format

Open
#6,600 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I wanted to be able to save a HF dataset for translations and load it again in another script, but I'm a bit confused with the documentation and the result I've got so I'm opening this issue to ask if this behavior is as expected.

### Steps to reproduce the bug

The documentation I've mainly consulted is https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/loading_methods#datasets.load_dataset and https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset (where I've found `.to_csv()`)

```python
# Load a dataset of translations
test_dataset = load_dataset("opus100", name="en-fr", split="test")

# Save with .to_csv()
test_csv_path = "try_testset_save.csv"
test_dataset.to_csv(test_csv_path)

# Load dataset from the CSV
loaded_dataset = load_dataset("csv", data_files=test_csv_path)
print(test_dataset_fromfile[0]["translation"])
print(test_dataset_fromfile[0]["translation"]["en"])
```
```
Creating CSV from Arrow format: 100%
2/2 [00:00<00:00, 47.99ba/s]
Downloading data files: 100%
1/1 [00:00<00:00, 65.33it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 42.10it/s]
Generating train split:
2000/0 [00:00<00:00, 47486.09 examples/s]

{'en': "She wasn't going to vaccinate her kid against polio, no way.", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[29], line 11
9 loaded_dataset = load_dataset("csv", data_files=test_csv_path)
10 print(test_dataset_fromfile[0]["translation"])
---> 11 print(test_dataset_fromfile[0]["translation"]["en"])

TypeError: string indices must be integers, not 'str'
```

### Expected behavior

Each translation was saved as a stringified dict like `"{'en': ""She wasn't going to vaccinate her kid against polio, no way."", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}"` where I would have expected 2 columns (1st with English segments, and 2nd with French segments), and I was expecting `load_dataset` to infer the type of feature automatically as I haven't seen anything about it in the documentation.

Do you have an example of how to effectively save and load datasets of translations ?

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.5
- `huggingface_hub` version: 0.16.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.