Document that `airflow db downgrade` from 3.2 to 3.1.x does not revert SDK-serialized metadata row content
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.1
### What happened and how to reproduce it?
We upgraded production from **Airflow 3.1.8** to **3.2.1**, hit scheduling issues, then attempted to roll back by:
1. Deploying Airflow **3.1.8** image
2. Running `airflow db downgrade -n 3.1.8`
Alembic migrations completed successfully. However, **scheduler and triggerer immediately crashed** on startup with:
```
KeyError:
```
in `BaseSerialization.deserialize`, while loading rows from the metadata database (e.g. during `_schedule_all_dag_runs` or trigger deserialization).
### Environment
- **Upgrade path:** 3.1.8 → 3.2.1 → attempted rollback to 3.1.8
- **Database:** PostgreSQL
- **Executor:** KubernetesExecutor
Example traceback (scheduler):
```
File ".../scheduler_job_runner.py", line 1968, in _schedule_all_dag_runs
callback_tuples = [(run, self._schedule_dag_run(run, session=session)) for run in dag_runs]
...
File ".../airflow/utils/sqlalchemy.py", line 137, in process_result_value
return BaseSerialization.deserialize(value)
File ".../airflow/serialization/serialized_objects.py", line 897, in deserialize
var = encoded_var[Encoding.VAR]
KeyError:
```
Similar failures occur in the triggerer when deserializing `trigger.encrypted_kwargs`.
### Root cause (our analysis)
There are **two separate layers** in the metadata DB:
| Layer | What downgrade migrations handle | What they do NOT handle |
|-------|-----------------------------------|-------------------------|
| Schema | Tables, columns, alembic revision | — |
| Row content | — | Serialized JSON blobs written while 3.2 was running |
While Airflow 3.2.x was running, it wrote metadata using **SDK serde** (see [#59711](https://github.com/apache/airflow/pull/59711) and 3.2.1 release notes — serde moved to `airflow.sdk.serde`). Examples:
- `trigger.encrypted_kwargs`
- `dag_run.conf` and related serialized columns
- Deferred task / trigger payloads
Airflow **3.1.8** reads these via legacy `BaseSerialization.deserialize()`, which expects the `{__type, __var}` wrapper format. SDK-serde blobs do not have `__var` at the top level → `KeyError`.
`airflow db downgrade` reverts the **schema** to 3.1.8-compatible structure but does **not** rewrite existing row payloads back to 3.1 serialization format.
### What we tried
- `airflow db downgrade -n 3.1.8` — succeeds, but runtime still crashes
- Deploying the old 3.1.8 application image — correct for code, insufficient for DB content
- Manual cleanup (risky): `DELETE FROM trigger;` + failing stuck `dag_run` rows — unblocks partially but is not a safe general solution
**Only clean rollback path:** restore PostgreSQL from a backup taken **before** the 3.2 upgrade.
### Expected behavior
The upgrade/downgrade documentation should clearly state:
1. **Downgrading Airflow major/minor versions is not fully supported** without a metadata DB backup/restore.
2. **`airflow db downgrade` only reverts schema** (alembic migrations). It does not migrate serialized row content.
3. After running 3.2.x against a database, rolling back to 3.1.x requires either:
- Restoring a pre-3.2 DB backup, or
- Manual cleanup of incompatible rows (triggers, active dag runs with 3.2-format conf, etc.) — with data loss risk
4. The 3.2 serde migration ([#59711](https://github.com/apache/airflow/pull/59711)) affects trigger kwargs and related fields; this is not reversed on downgrade.
Suggested doc locations:
- Upgrade guide / release notes for 3.2.0 / 3.2.1
- `docs/howto/upgrading.rst` or equivalent
- `airflow db downgrade` CLI help text
### Actual behavior
- Downgrade migrations report success
- Users reasonably assume DB is compatible with 3.1.8
- Scheduler/triggerer crashloop with opaque `KeyError: __var`
- No guidance on which tables/rows are affected or how to recover
### Suggested diagnostic queries
```sql
-- Triggers written under 3.2 SDK serde (may lack __var wrapper)
SELECT id, classpath, LEFT(encrypted_kwargs::text, 120)
FROM trigger
LIMIT 20;
-- Active dag runs that may carry 3.2-format conf
SELECT dag_id, run_id, state, LEFT(conf::text, 120)
FROM dag_run
WHERE state IN ('running', 'queued')
AND conf IS NOT NULL
AND conf::text NOT LIKE '%__var%';
```
### Related issues / PRs
- [#59711](https://github.com/apache/airflow/pull/59711) — SDK serde for trigger/next kwargs
- [#64613](https://github.com/apache/airflow/issues/64613) — trigger deserialization errors with external-event DAGs
- [#65973](https://github.com/apache/airflow/issues/65973) — asset trigger kwargs format change 3.1.8 → 3.2.1
- [#65688](https://github.com/apache/airflow/pull/65688) — scheduler UniqueViolation on downgrade 3.2.0 → 3.1.x (schema-level fix, not serde data)
- [#63434](https://github.com/apache/airflow/issues/63434), [#63444](https://github.com/apache/airflow/issues/63444), [#63535](https://github.com/apache/airflow/issues/63535) — other 3.2 → 3.1 downgrade migration failures
### Why this matters
Teams hitting issues on 3.2 may attempt downgrade as first recovery step. Schema-successful downgrade with runtime failure is worse than a clear "unsupported — restore from backup" message. We lost time debugging this as a dependency/version mismatch before identifying the serde data layer.
### What you think should happen instead?
_No response_
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.