huggingface / huggingface/datasets
Json builder: Allow features to override problematic Arrow types
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
In the JSON builder, use explicitly requested feature types before or while converting to Arrow.
### Motivation
Working with JSON datasets is really hard because of Arrow. At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types. But it seems like this is not possible because the features are only used *after* converting to arrow.
Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing
### Your contribution
Maybe with some guidance. I'm not very familiar with arrow or pandas.
Contributor guide
Assessment
This issue has not been assessed yet.