huggingface / huggingface/datasets

Custom features not compatible with special encoding/decoding logic

Open
#7,220 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

It is possible to register custom features using datasets.features.features.register_feature (https://github.com/huggingface/datasets/pull/6727)

However such features are not compatible with Features.encode_example/decode_example if they require special encoding / decoding logic because encode_nested_example / decode_nested_example checks whether the feature is in a fixed list of encodable types:

https://github.com/huggingface/datasets/blob/16a121d7821a7691815a966270f577e2c503473f/src/datasets/features/features.py#L1349

This prevents the extensibility of features to complex cases

### Steps to reproduce the bug

```python
class ListOfStrs:
def encode_example(self, value):
if isinstance(value, str):
return [str]
else:
return value
feats = Features(strlist=ListOfStrs())
assert feats.encode_example({"strlist": "a"})["strlist"] = feats["strlist"].encode_example("a")}
```

### Expected behavior

Registered feature types should be encoded based on some property of the feature (e.g. requires_encoding)?

### Environment info

3.0.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.