huggingface / huggingface/datasets
to_json cannot round-trip temporal columns: all timestamps written as epoch milliseconds
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`Dataset.to_json()` serializes every temporal column as a bare epoch integer at a single fixed resolution (pandas' `date_unit="ms"` default), with no unit recorded in the output. Timestamps therefore do not survive a `to_json` → `from_json` round trip: they either come back silently wrong or raise.
### Steps to reproduce the bug
```python
import datetime, tempfile
from datasets import Dataset, Features, Value
for unit, val in [("s", datetime.datetime(2024,1,1,12,30,45)),
("ms", datetime.datetime(2024,1,1,12,30,45,123000)),
("us", datetime.datetime(2024,1,1,12,30,45,123456))]:
f = Features({"t": Value(f"timestamp[{unit}]")})
ds = Dataset.from_dict({"t": [val]}, features=f)
p = tempfile.mktemp(suffix=".jsonl")
ds.to_json(p)
print(unit, open(p).read().strip(), end=" ")
try:
print(Dataset.from_json(p, features=f)[0]["t"])
except Exception as e:
print(type(e).__name__, e)
```
```
s {"t":1704112245000} OverflowError date value out of range
ms {"t":1704112245123} 2024-01-01 12:30:45.123000 # only this one works
us {"t":1704112245123} 1970-01-20 17:21:52.245123 # silently wrong
```
The `timestamp[us]` case is the most dangerous: no error, and the value comes back 54 years off. `timestamp[us]` and `[ns]` also silently lose sub-millisecond precision on write. Reading back without explicit `features` yields `Value('int64')` for all three.
### Cause
`src/datasets/io/json.py` calls `batch.to_json(...)` without setting `date_format` / `date_unit`, so pandas' `date_unit="ms"` default is applied uniformly to every temporal column regardless of its Arrow unit.
### Notes for whoever picks this up
There is no single `date_unit` value that fixes this, because pandas applies one global setting to all temporal columns:
- `date_unit="us"` fixes `[us]` but breaks `[s]` (`OverflowError: Python int too large to convert to C int`) and `[ms]` (`OverflowError: date value out of range`) — both of which behave acceptably today.
- `date_format="iso"` makes the value self-describing and fixes `[us]`, but `[s]` and `[ms]` then fail to load with `DatasetGenerationError`, and inference reads the column back as `Value('string')`.
So it likely needs the writer to derive the unit/format per column from its Arrow type, and possibly the JSON loader to recognise ISO timestamps. Worth noting that changing the on-disk format affects every existing JSON export, so this may warrant an opt-in or a version-gated default.
Happy to open a PR if you can point me at the direction you'd prefer.
### Environment info
- `datasets` 5.0.2.dev0 (`main` @ b7cb10b0e)
- pyarrow 25.0.0, pandas 2.x, Python 3.12
Contributor guide
Research direction
Start in src/datasets/io/json.py, where Dataset.to_json calls batch.to_json, and inspect the corresponding Dataset.from_json loading path. Reproduce the three timestamp cases from the issue, then determine how writer and loader changes can preserve each Arrow unit and precision without breaking existing JSON exports; done means all cases round-trip correctly and inference remains appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100