apache / apache/airflow

Document that `airflow db downgrade` from 3.2 to 3.1.x does not revert SDK-serialized metadata row content

Open
#68,317 0 comments 0 reactions 0 assignees View on GitHub
area:upgrade kind:bug kind:documentation needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.1

### What happened and how to reproduce it?

We upgraded production from **Airflow 3.1.8** to **3.2.1**, hit scheduling issues, then attempted to roll back by:

1. Deploying Airflow **3.1.8** image
2. Running `airflow db downgrade -n 3.1.8`

Alembic migrations completed successfully. However, **scheduler and triggerer immediately crashed** on startup with:

```
KeyError:
```

in `BaseSerialization.deserialize`, while loading rows from the metadata database (e.g. during `_schedule_all_dag_runs` or trigger deserialization).

### Environment

- **Upgrade path:** 3.1.8 → 3.2.1 → attempted rollback to 3.1.8
- **Database:** PostgreSQL
- **Executor:** KubernetesExecutor

Example traceback (scheduler):

```
File ".../scheduler_job_runner.py", line 1968, in _schedule_all_dag_runs
callback_tuples = [(run, self._schedule_dag_run(run, session=session)) for run in dag_runs]
...
File ".../airflow/utils/sqlalchemy.py", line 137, in process_result_value
return BaseSerialization.deserialize(value)
File ".../airflow/serialization/serialized_objects.py", line 897, in deserialize
var = encoded_var[Encoding.VAR]
KeyError:
```

Similar failures occur in the triggerer when deserializing `trigger.encrypted_kwargs`.

### Root cause (our analysis)

There are **two separate layers** in the metadata DB:

| Layer | What downgrade migrations handle | What they do NOT handle |
|-------|-----------------------------------|-------------------------|
| Schema | Tables, columns, alembic revision | — |
| Row content | — | Serialized JSON blobs written while 3.2 was running |

While Airflow 3.2.x was running, it wrote metadata using **SDK serde** (see [#59711](https://github.com/apache/airflow/pull/59711) and 3.2.1 release notes — serde moved to `airflow.sdk.serde`). Examples:

- `trigger.encrypted_kwargs`
- `dag_run.conf` and related serialized columns
- Deferred task / trigger payloads

Airflow **3.1.8** reads these via legacy `BaseSerialization.deserialize()`, which expects the `{__type, __var}` wrapper format. SDK-serde blobs do not have `__var` at the top level → `KeyError`.

`airflow db downgrade` reverts the **schema** to 3.1.8-compatible structure but does **not** rewrite existing row payloads back to 3.1 serialization format.

### What we tried

- `airflow db downgrade -n 3.1.8` — succeeds, but runtime still crashes
- Deploying the old 3.1.8 application image — correct for code, insufficient for DB content
- Manual cleanup (risky): `DELETE FROM trigger;` + failing stuck `dag_run` rows — unblocks partially but is not a safe general solution

**Only clean rollback path:** restore PostgreSQL from a backup taken **before** the 3.2 upgrade.

### Expected behavior

The upgrade/downgrade documentation should clearly state:

1. **Downgrading Airflow major/minor versions is not fully supported** without a metadata DB backup/restore.
2. **`airflow db downgrade` only reverts schema** (alembic migrations). It does not migrate serialized row content.
3. After running 3.2.x against a database, rolling back to 3.1.x requires either:
- Restoring a pre-3.2 DB backup, or
- Manual cleanup of incompatible rows (triggers, active dag runs with 3.2-format conf, etc.) — with data loss risk
4. The 3.2 serde migration ([#59711](https://github.com/apache/airflow/pull/59711)) affects trigger kwargs and related fields; this is not reversed on downgrade.

Suggested doc locations:

- Upgrade guide / release notes for 3.2.0 / 3.2.1
- `docs/howto/upgrading.rst` or equivalent
- `airflow db downgrade` CLI help text

### Actual behavior

- Downgrade migrations report success
- Users reasonably assume DB is compatible with 3.1.8
- Scheduler/triggerer crashloop with opaque `KeyError: __var`
- No guidance on which tables/rows are affected or how to recover

### Suggested diagnostic queries

```sql
-- Triggers written under 3.2 SDK serde (may lack __var wrapper)
SELECT id, classpath, LEFT(encrypted_kwargs::text, 120)
FROM trigger
LIMIT 20;

-- Active dag runs that may carry 3.2-format conf
SELECT dag_id, run_id, state, LEFT(conf::text, 120)
FROM dag_run
WHERE state IN ('running', 'queued')
AND conf IS NOT NULL
AND conf::text NOT LIKE '%__var%';
```

### Related issues / PRs

- [#59711](https://github.com/apache/airflow/pull/59711) — SDK serde for trigger/next kwargs
- [#64613](https://github.com/apache/airflow/issues/64613) — trigger deserialization errors with external-event DAGs
- [#65973](https://github.com/apache/airflow/issues/65973) — asset trigger kwargs format change 3.1.8 → 3.2.1
- [#65688](https://github.com/apache/airflow/pull/65688) — scheduler UniqueViolation on downgrade 3.2.0 → 3.1.x (schema-level fix, not serde data)
- [#63434](https://github.com/apache/airflow/issues/63434), [#63444](https://github.com/apache/airflow/issues/63444), [#63535](https://github.com/apache/airflow/issues/63535) — other 3.2 → 3.1 downgrade migration failures

### Why this matters

Teams hitting issues on 3.2 may attempt downgrade as first recovery step. Schema-successful downgrade with runtime failure is worse than a clear "unsupported — restore from backup" message. We lost time debugging this as a dependency/version mismatch before identifying the serde data layer.

### What you think should happen instead?

_No response_

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.