huggingface / huggingface/datasets

to_json cannot round-trip temporal columns: all timestamps written as epoch milliseconds

Open
#8,390 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`Dataset.to_json()` serializes every temporal column as a bare epoch integer at a single fixed resolution (pandas' `date_unit="ms"` default), with no unit recorded in the output. Timestamps therefore do not survive a `to_json` → `from_json` round trip: they either come back silently wrong or raise.

### Steps to reproduce the bug

```python
import datetime, tempfile
from datasets import Dataset, Features, Value

for unit, val in [("s", datetime.datetime(2024,1,1,12,30,45)),
("ms", datetime.datetime(2024,1,1,12,30,45,123000)),
("us", datetime.datetime(2024,1,1,12,30,45,123456))]:
f = Features({"t": Value(f"timestamp[{unit}]")})
ds = Dataset.from_dict({"t": [val]}, features=f)
p = tempfile.mktemp(suffix=".jsonl")
ds.to_json(p)
print(unit, open(p).read().strip(), end=" ")
try:
print(Dataset.from_json(p, features=f)[0]["t"])
except Exception as e:
print(type(e).__name__, e)
```

```
s {"t":1704112245000} OverflowError date value out of range
ms {"t":1704112245123} 2024-01-01 12:30:45.123000 # only this one works
us {"t":1704112245123} 1970-01-20 17:21:52.245123 # silently wrong
```

The `timestamp[us]` case is the most dangerous: no error, and the value comes back 54 years off. `timestamp[us]` and `[ns]` also silently lose sub-millisecond precision on write. Reading back without explicit `features` yields `Value('int64')` for all three.

### Cause

`src/datasets/io/json.py` calls `batch.to_json(...)` without setting `date_format` / `date_unit`, so pandas' `date_unit="ms"` default is applied uniformly to every temporal column regardless of its Arrow unit.

### Notes for whoever picks this up

There is no single `date_unit` value that fixes this, because pandas applies one global setting to all temporal columns:

- `date_unit="us"` fixes `[us]` but breaks `[s]` (`OverflowError: Python int too large to convert to C int`) and `[ms]` (`OverflowError: date value out of range`) — both of which behave acceptably today.
- `date_format="iso"` makes the value self-describing and fixes `[us]`, but `[s]` and `[ms]` then fail to load with `DatasetGenerationError`, and inference reads the column back as `Value('string')`.

So it likely needs the writer to derive the unit/format per column from its Arrow type, and possibly the JSON loader to recognise ISO timestamps. Worth noting that changing the on-disk format affects every existing JSON export, so this may warrant an opt-in or a version-gated default.

Happy to open a PR if you can point me at the direction you'd prefer.

### Environment info

- `datasets` 5.0.2.dev0 (`main` @ b7cb10b0e)
- pyarrow 25.0.0, pandas 2.x, Python 3.12

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/io/json.py, where Dataset.to_json calls batch.to_json, and inspect the corresponding Dataset.from_json loading path. Reproduce the three timestamp cases from the issue, then determine how writer and loader changes can preserve each Arrow unit and precision without breaking existing JSON exports; done means all cases round-trip correctly and inference remains appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.