huggingface / huggingface/datasets

Lossy json serialization - deserialization of dataset info

Open
#5,288 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead.

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
dataset = load_dataset(d, split="train")
dataset.to_json('_test')
dataset_loaded = load_dataset("json", data_files='_test', split='train')
try:
assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
except Exception as ex:
print(f'{ex}')
print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')

test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)},
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```

### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.