huggingface / huggingface/datasets
Allow passing a subset of output features to Dataset.map
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
Currently, map does one of two things to the features (if I'm not mistaken):
* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this
However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.
### Motivation
To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings
Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```
The result is a float again, since after the mapping operation it forces the old datatypes back on the data.
Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
return {**tokenizer(r["text"]), "y": int(r["y"])}
data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
format_data,
features=Features({'y': Value(dtype="int64")}),
remove_columns=["text"],
)
```
Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.
### Your contribution
I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.
Contributor guide
Assessment
This issue has not been assessed yet.