huggingface / huggingface/datasets
mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`
### Steps to reproduce the bug
```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D
print(f"datasets version: {datasets.__version__}")
data_info = {
"arr_float" : "float64",
"arr_int" : "int32"
}
sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}
features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)
dataset = Dataset.from_dict(sample, features=features)
ds = dataset.with_format("numpy")
for key in features:
print(f"{key} feature dtype: ", ds.features[key].dtype)
print(f"{key} dtype:", ds[key].dtype)
```
Output:
```bash
datasets version: 3.0.2
arr_float feature dtype: float64
arr_float dtype: float32
arr_int feature dtype: int32
arr_int dtype: int64
```
### Expected behavior
It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value
### Environment info
- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
Contributor guide
Assessment
This issue has not been assessed yet.