huggingface / huggingface/datasets

Empty column loses its dtype under numpy format (always float32)

Open
#8,469 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

Under `numpy` format, a column that holds zero rows comes back as `float32` no matter what dtype it was declared with. The same column keeps its dtype as soon as it holds a single row, and the `pandas` and `arrow` formatters keep it in both cases.

This is easy to hit in normal code: any `filter` that matches nothing, or `select([])`.

### Steps or code to reproduce the bug

```python
import numpy as np
from datasets import Dataset, Features, Value

feats = Features({"a": Value("int64")})
ds = Dataset.from_dict({"a": [1, 2, 3]}, features=feats).with_format("numpy")

matched = ds.filter(lambda x: x["a"] > 1)[:]["a"]
unmatched = ds.filter(lambda x: x["a"] > 99)[:]["a"]

print(matched.dtype) # int64
print(unmatched.dtype) # float32
```

Across dtypes:

| declared | non-empty | empty |
|---|---|---|
| `int64` | `int64` | **`float32`** |
| `int32` | `int64` | **`float32`** |
| `float64` | `float32` | `float32` |
| `bool` | `bool` | **`float32`** |
| `string` | ` int64` and `float64 -> float32` on the non-empty rows are the formatter's documented `default_dtype` behaviour and not part of this report.

### Expected behavior

An empty column should keep the dtype the same column has when non-empty, as it already does under `pandas` and `arrow`.

PR: #8470.

### Environment info

- `datasets` version: 5.0.2.dev0 (`main` @ 48b7ee7)
- Python version: 3.11.9
- Platform: Windows 11
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- NumPy version: 2.4.6

Contributor guide

Open the contributing guide

Research direction

Read src/datasets/formatting/formatting.py around NumpyArrowExtractor._arrow_array_to_numpy and NumpyFormatter._tensorize, then run the empty-column reproduction from the issue. Done means empty numpy-formatted columns retain their declared dtype, including non-numeric columns, without changing the documented non-empty default_dtype behavior; PR #8470 indicates work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.