huggingface / huggingface/datasets
with_format("numpy") returns bool/datetime values as 0-d unhashable arrays and casts duration columns to int64
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
With numpy formatting, row access returns `np.int64`/`np.float64`/`np.str_` scalars for most dtypes, but **bool and timestamp/date values come back as 0-dimensional `ndarray`s** (`array(True)`), which are unhashable and fail `isinstance(x, np.bool_)` checks. Separately, **batch/column access silently casts `duration` columns from `timedelta64[us]` to plain `int64`** (row access keeps `np.timedelta64`), so the same column reports different types depending on how it's read.
Root cause: `NumpyFormatter._tensorize` (`src/datasets/formatting/np_formatter.py`) early-returns only `np.number` scalars — `np.bool_`/`np.datetime64` aren't `np.number`, so they fall through to `np.asarray()` (and `_recursive_tensorize` re-wraps them via `__array__` first). The int64 default-dtype branch uses `np.issubdtype(dtype, np.integer)`, which is True for `timedelta64` in numpy's type hierarchy, so duration arrays get the int64 default.
### Steps to reproduce the bug
```python
from datasets import Dataset
import datetime as dt
ds = Dataset.from_dict({'b': [True], 't': [dt.datetime(2024,1,1)],
'td': [dt.timedelta(seconds=5)]}).with_format('numpy')
row = ds[0]
# actual: b -> array(True) (ndarray!), t -> array('2024-01-01...') (ndarray!)
# td -> np.timedelta64(5000000,'us') (scalar, inconsistent with batch below)
hash(row['b']) # TypeError: unhashable type: 'numpy.ndarray'
print(ds[:]['td'].dtype) # int64 <- duration unit silently stripped
```
### Expected behavior
`np.True_`, `np.datetime64(...)` scalars for row access (consistent with int/float/str), and `ds[:]['td'].dtype == timedelta64[us]` (consistent between row and batch access).
I have a fix ready (7-line change + regression tests) and will open a PR.
### Environment info
datasets main (5.0.2.dev0), pyarrow 25.0.1, numpy 2.4.6, Python 3.11.14, macOS
Contributor guide
Research direction
Start in src/datasets/formatting/np_formatter.py, focusing on NumpyFormatter._tensorize and the int64 default-dtype branch. Run the reproduction in the issue and add regression coverage for row bool/datetime scalars and batch duration dtype. Done means row access returns NumPy scalar values and duration remains timedelta64[us] for batch access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100