huggingface / huggingface/datasets
Support for data with instance-wise dictionary as features
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.
It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).
### Motivation
I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
"index": 0,
"feature": {
"2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
...
}
},
...
{
"index": 9999,
"feature": {
"x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
...
}
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
"index": 0,
"feature": {
"2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
...
"x >= 6": None, # keys from other instances
...
}
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.
A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.
### Your contribution
N/A
Contributor guide
Assessment
This issue has not been assessed yet.