huggingface / huggingface/datasets
Empty column loses its dtype under numpy format (always float32)
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Under `numpy` format, a column that holds zero rows comes back as `float32` no matter what dtype it was declared with. The same column keeps its dtype as soon as it holds a single row, and the `pandas` and `arrow` formatters keep it in both cases.
This is easy to hit in normal code: any `filter` that matches nothing, or `select([])`.
### Steps or code to reproduce the bug
```python
import numpy as np
from datasets import Dataset, Features, Value
feats = Features({"a": Value("int64")})
ds = Dataset.from_dict({"a": [1, 2, 3]}, features=feats).with_format("numpy")
matched = ds.filter(lambda x: x["a"] > 1)[:]["a"]
unmatched = ds.filter(lambda x: x["a"] > 99)[:]["a"]
print(matched.dtype) # int64
print(unmatched.dtype) # float32
```
Across dtypes:
| declared | non-empty | empty |
|---|---|---|
| `int64` | `int64` | **`float32`** |
| `int32` | `int64` | **`float32`** |
| `float64` | `float32` | `float32` |
| `bool` | `bool` | **`float32`** |
| `string` | ` int64` and `float64 -> float32` on the non-empty rows are the formatter's documented `default_dtype` behaviour and not part of this report.
### Expected behavior
An empty column should keep the dtype the same column has when non-empty, as it already does under `pandas` and `arrow`.
PR: #8470.
### Environment info
- `datasets` version: 5.0.2.dev0 (`main` @ 48b7ee7)
- Python version: 3.11.9
- Platform: Windows 11
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- NumPy version: 2.4.6
Contributor guide
Research direction
Read src/datasets/formatting/formatting.py around NumpyArrowExtractor._arrow_array_to_numpy and NumpyFormatter._tensorize, then run the empty-column reproduction from the issue. Done means empty numpy-formatted columns retain their declared dtype, including non-numeric columns, without changing the documented non-empty default_dtype behavior; PR #8470 indicates work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100