apache / apache/airflow

Naive datetime in XCom changes value when workers have different timezones

Open
#72,635 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

main (3.4.0, `123b1be658`)

### What happened and how to reproduce it?

A naive `datetime` pushed to XCom by one task comes back with a different value when the task that pulls it runs on a worker with a different OS timezone. Nothing errors; the value is just wrong.

`serialize` in `task-sdk/src/airflow/sdk/serde/serializers/datetime.py` stores `o.timestamp()`, which interprets a naive datetime as the **writing** process's local time. `deserialize` calls `datetime.fromtimestamp(ts, tz=None)`, which renders it in the **reading** process's local time. The two are the same machine in a single-process test and different machines in most real deployments.

```python
# task A, worker with TZ=UTC
from airflow.sdk.bases.xcom import BaseXCom
import datetime
blob = BaseXCom.serialize_value(datetime.datetime(2026, 1, 1, 12, 0, 0))

# task B, worker with TZ=America/New_York
import types
BaseXCom.deserialize_value(types.SimpleNamespace(value=blob))
```

Running that split three ways on `main`:

```
-- control: both workers UTC
task A on TZ=UTC pushed 2026-01-01T12:00:00
task B on TZ=UTC pulled 2026-01-01T12:00:00
identical: True

-- worker A UTC, worker B America/New_York
task A on TZ=UTC pushed 2026-01-01T12:00:00
task B on TZ=America/New_York pulled 2026-01-01T07:00:00
identical: False

-- worker A UTC, worker B Asia/Kolkata
task A on TZ=UTC pushed 2026-01-01T12:00:00
task B on TZ=Asia/Kolkata pulled 2026-01-01T17:30:00
identical: False
```

A timezone-aware datetime survives the same split unchanged, because the tz is stored alongside the timestamp. `datetime.date` is also unaffected — it is serialized with `isoformat()` rather than an epoch. So this is specific to naive `datetime`.

### What you think should happen instead?

A naive datetime should come back equal to what went in, and the interpretation should not depend on which process reads it.

The timezone docs already define what a naive datetime means in Airflow:

> In case a naive `start_date` or `end_date` is encountered the default time zone is applied. It is applied in such a way that it is assumed that the naive date time is already in the default time zone.

So the intended anchor is `core.default_timezone`, not the OS local time of whichever worker happens to deserialize. The current behaviour is neither: it is the writer's OS zone on the way in and the reader's on the way out.

I did look for a documented "naive datetimes are unsupported in XCom" rule before filing and could not find one; the paragraph above is the only statement of intent I found, and it points the other way.

### Anything else?

The awkward part is the serialized form rather than the fix. `{"timestamp": ..., "tz": null}` cannot express "naive" — an epoch is a fixed instant, and naive means the opposite. Any correction changes the payload, so it needs `__version__` bumped to 3 with the version-2 shape still readable, and XComs persist in the database across upgrades. That is a call for maintainers rather than something to pick unilaterally, but the options seem to be:

1. Serialize naive datetimes as an ISO wall-clock string, the way `date` already is, and keep the `{timestamp, tz}` form for aware ones. Round-trips exactly and needs no timezone config at read time.
2. Anchor naive datetimes to `core.default_timezone` at serialize time and store the resulting aware instant, matching the documented interpretation. Changes value if the setting changes between write and read.
3. Store the epoch as now, plus an explicit `naive: true` flag and the wall clock, so old readers keep working.

Happy to implement whichever direction you prefer.

### Are you willing to submit PR?

Yes, once there is a view on which of the above you want.

Contributor guide

Open the contributing guide

Research direction

Start with task-sdk/src/airflow/sdk/serde/serializers/datetime.py and the BaseXCom serialize_value/deserialize_value path, then run the reported reproduction with different OS timezones. Review the timezone documentation and existing versioned serialization behavior. Done requires a maintainer-selected representation, version-2 compatibility, and tests covering naive and aware datetimes across worker timezones.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.