huggingface / huggingface/datasets
Support cast_kwargs in cast_columns
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
expose `cast(**cast_kwargs)` to `cast_column()`
https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2205
### Motivation
`cast_column()` wraps `cast()` function without exposing any `cast()` args. For large multi-modal datasets, e.g.
```py
# a dataset with list[{"bytes"}: b'', ...], much more than one image
load_dataset("MLLM-CL/VTCBench").cast_column("images", List(Image(decode=False)))
```
This would fail due to #6206, #7167, where the default value `1000` for batch size in `cast()` is too large and causes `pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays`.
https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2164-L2205
### Your contribution
#7910
Contributor guide
Assessment
This issue has not been assessed yet.