huggingface / huggingface/datasets

with_format("numpy") returns bool/datetime values as 0-d unhashable arrays and casts duration columns to int64

Open
#8,500 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

With numpy formatting, row access returns `np.int64`/`np.float64`/`np.str_` scalars for most dtypes, but **bool and timestamp/date values come back as 0-dimensional `ndarray`s** (`array(True)`), which are unhashable and fail `isinstance(x, np.bool_)` checks. Separately, **batch/column access silently casts `duration` columns from `timedelta64[us]` to plain `int64`** (row access keeps `np.timedelta64`), so the same column reports different types depending on how it's read.

Root cause: `NumpyFormatter._tensorize` (`src/datasets/formatting/np_formatter.py`) early-returns only `np.number` scalars — `np.bool_`/`np.datetime64` aren't `np.number`, so they fall through to `np.asarray()` (and `_recursive_tensorize` re-wraps them via `__array__` first). The int64 default-dtype branch uses `np.issubdtype(dtype, np.integer)`, which is True for `timedelta64` in numpy's type hierarchy, so duration arrays get the int64 default.

### Steps to reproduce the bug

```python
from datasets import Dataset
import datetime as dt
ds = Dataset.from_dict({'b': [True], 't': [dt.datetime(2024,1,1)],
'td': [dt.timedelta(seconds=5)]}).with_format('numpy')
row = ds[0]
# actual: b -> array(True) (ndarray!), t -> array('2024-01-01...') (ndarray!)
# td -> np.timedelta64(5000000,'us') (scalar, inconsistent with batch below)
hash(row['b']) # TypeError: unhashable type: 'numpy.ndarray'
print(ds[:]['td'].dtype) # int64 <- duration unit silently stripped
```

### Expected behavior

`np.True_`, `np.datetime64(...)` scalars for row access (consistent with int/float/str), and `ds[:]['td'].dtype == timedelta64[us]` (consistent between row and batch access).

I have a fix ready (7-line change + regression tests) and will open a PR.

### Environment info

datasets main (5.0.2.dev0), pyarrow 25.0.1, numpy 2.4.6, Python 3.11.14, macOS

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/formatting/np_formatter.py, focusing on NumpyFormatter._tensorize and the int64 default-dtype branch. Run the reproduction in the issue and add regression coverage for row bool/datetime scalars and batch duration dtype. Done means row access returns NumPy scalar values and duration remains timedelta64[us] for batch access.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.