huggingface / huggingface/datasets
Lossy json serialization - deserialization of dataset info
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead.
### Steps to reproduce the bug
```
from datasets import load_dataset
def test_serdes_from_json(d):
dataset = load_dataset(d, split="train")
dataset.to_json('_test')
dataset_loaded = load_dataset("json", data_files='_test', split='train')
try:
assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
except Exception as ex:
print(f'{ex}')
print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
test_serdes_from_json('rotten_tomatoes')
```
Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)},
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```
### Expected behavior
The deserialized `features.label` should have type `ClassLabel`.
### Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3
Contributor guide
Assessment
This issue has not been assessed yet.