huggingface / huggingface/datasets
Nanosecond timestamps are silently truncated even when Value("timestamp[ns]") is requested
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`Dataset.from_dict` / `from_list` / `map` silently drop sub-microsecond precision, even when the user explicitly declares a nanosecond feature type. The resulting dataset reports `timestamp[ns]` but the data has already been truncated to microseconds.
### Steps to reproduce the bug
```python
import pandas as pd
from datasets import Dataset, Features, Value
ts = pd.Timestamp("2024-01-01 00:00:00.123456789")
ds = Dataset.from_dict({"t": [ts]}, features=Features({"t": Value("timestamp[ns]")}))
print(ds.features) # {'t': Value('timestamp[ns]')}
print(ds[0]["t"]) # 2024-01-01 00:00:00.123456 <-- .789 ns lost
```
The same applies to `Value("duration[ns]")` via `pd.Timedelta`. `Dataset.from_pandas` on a `datetime64[ns]` column preserves the nanoseconds, so the two entry points disagree.
### Cause
Two compounding steps:
1. `src/datasets/features/features.py` converts `pd.Timestamp` → `datetime.datetime` via `to_pydatetime()`, which only has microsecond resolution (pandas even emits `UserWarning: Discarding nonzero nanoseconds in conversion`).
2. More fundamentally, `src/datasets/arrow_writer.py` calls `pa.array(cast_to_python_objects(examples, only_1d_for_numpy=True))` **without passing the target type**. PyArrow infers `timestamp[us]` from any datetime object, so the data is truncated before `cast_array_to_feature` casts the already-lossy array up to `timestamp[ns]`.
### Notes for whoever picks this up
I tried the obvious narrow fixes and neither is safe, so I'm filing rather than sending a patch:
- **Skipping `to_pydatetime()` when `obj.nanosecond != 0` changes nothing user-visible.** Every caller feeds the result straight into `pa.array(...)` with no type, so pyarrow re-infers `timestamp[us]` and truncates anyway. The only effect is suppressing the pandas warning.
- **Returning `obj.to_datetime64()`** does make pyarrow infer `timestamp[ns]` losslessly, but it is not a safe drop-in: `_cast_to_python_objects` decides list-wide conversion from the first element only, so a column mixing nanosecond and non-nanosecond timestamps produces a mixed `[np.datetime64, datetime.datetime]` list and pyarrow raises `ArrowInvalid: numpy.datetime64 scalars cannot be mixed with other Python scalar values`. That turns a silent truncation into a crash. Making it uniform requires scanning whole columns, which is a perf cost on the hottest write path.
The real fix is probably to thread the known `pa_type` into `pa.array()` in `TypedSequence._arrow_array`, which is a core-writer change the maintainers should own.
Related: #8390 covers a separate temporal-precision problem in `to_json`.
### Environment info
- `datasets` 5.0.2.dev0 (`main` @ b7cb10b0e)
- pyarrow 25.0.0, Python 3.12
Contributor guide
Research direction
Start in src/datasets/arrow_writer.py at TypedSequence._arrow_array and trace how Dataset.from_dict, from_list, and map build Arrow arrays from declared features. Compare this with the conversions in src/datasets/features/features.py and reproduce both timestamp[ns] and duration[ns] cases, including mixed-precision values. Done means sub-microsecond precision is preserved consistently without breaking mixed timestamp columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100