huggingface / huggingface/datasets

Nanosecond timestamps are silently truncated even when Value("timestamp[ns]") is requested

Open
#8,391 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`Dataset.from_dict` / `from_list` / `map` silently drop sub-microsecond precision, even when the user explicitly declares a nanosecond feature type. The resulting dataset reports `timestamp[ns]` but the data has already been truncated to microseconds.

### Steps to reproduce the bug

```python
import pandas as pd
from datasets import Dataset, Features, Value

ts = pd.Timestamp("2024-01-01 00:00:00.123456789")
ds = Dataset.from_dict({"t": [ts]}, features=Features({"t": Value("timestamp[ns]")}))
print(ds.features) # {'t': Value('timestamp[ns]')}
print(ds[0]["t"]) # 2024-01-01 00:00:00.123456 <-- .789 ns lost
```

The same applies to `Value("duration[ns]")` via `pd.Timedelta`. `Dataset.from_pandas` on a `datetime64[ns]` column preserves the nanoseconds, so the two entry points disagree.

### Cause

Two compounding steps:

1. `src/datasets/features/features.py` converts `pd.Timestamp` → `datetime.datetime` via `to_pydatetime()`, which only has microsecond resolution (pandas even emits `UserWarning: Discarding nonzero nanoseconds in conversion`).
2. More fundamentally, `src/datasets/arrow_writer.py` calls `pa.array(cast_to_python_objects(examples, only_1d_for_numpy=True))` **without passing the target type**. PyArrow infers `timestamp[us]` from any datetime object, so the data is truncated before `cast_array_to_feature` casts the already-lossy array up to `timestamp[ns]`.

### Notes for whoever picks this up

I tried the obvious narrow fixes and neither is safe, so I'm filing rather than sending a patch:

- **Skipping `to_pydatetime()` when `obj.nanosecond != 0` changes nothing user-visible.** Every caller feeds the result straight into `pa.array(...)` with no type, so pyarrow re-infers `timestamp[us]` and truncates anyway. The only effect is suppressing the pandas warning.
- **Returning `obj.to_datetime64()`** does make pyarrow infer `timestamp[ns]` losslessly, but it is not a safe drop-in: `_cast_to_python_objects` decides list-wide conversion from the first element only, so a column mixing nanosecond and non-nanosecond timestamps produces a mixed `[np.datetime64, datetime.datetime]` list and pyarrow raises `ArrowInvalid: numpy.datetime64 scalars cannot be mixed with other Python scalar values`. That turns a silent truncation into a crash. Making it uniform requires scanning whole columns, which is a perf cost on the hottest write path.

The real fix is probably to thread the known `pa_type` into `pa.array()` in `TypedSequence._arrow_array`, which is a core-writer change the maintainers should own.

Related: #8390 covers a separate temporal-precision problem in `to_json`.

### Environment info

- `datasets` 5.0.2.dev0 (`main` @ b7cb10b0e)
- pyarrow 25.0.0, Python 3.12

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/arrow_writer.py at TypedSequence._arrow_array and trace how Dataset.from_dict, from_list, and map build Arrow arrays from declared features. Compare this with the conversions in src/datasets/features/features.py and reproduce both timestamp[ns] and duration[ns] cases, including mixed-precision values. Done means sub-microsecond precision is preserved consistently without breaking mixed timestamp columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.