apache / apache/airflow

Downgrading DB leaves instance unable to clear tasks

Open
#57,747 6 comments 0 reactions 0 assignees View on GitHub
area:core area:db-migrations kind:bug priority:medium
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.11.0

### If "Other Airflow 2/3 version" selected, which one?

_No response_

### What happened?

Recently I accidentally applied DB migrations 28 through 68 while attempting to update from Airflow 2.10.5 to 2.11.0. After cleaning up the mess by downgrading the DB I noticed that I'm no longer able to clear tasks in order to re-try them.

Trying gives you nothing except a lengthy exception in the `webserver` logs, the relevant part of which is this:
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "id" of relation "task_instance_history" violates not-null constraint
DETAIL: Failing row contains (nosify.notify_successes, cert_env, manual__2025-11-02T23:00:04.866429+00:00, 0, 1, 2025-11-02 23:04:02.810671+00, 2025-11-02 23:04:05.093361+00, 2.28269, success,
0, airflow-worker-0.airflow-worker.airflow.svc.cluster.local, airflow, default_pool, 1, default, 1, _PythonDecoratedOperator, null, 2025-11-02 23:04:01.165684+00, 720119, 68415, null, \x80057d942e, 2025
-11-02 23:04:05.146802+00, null, a96e4277-0d0e-4b53-868d-48e72e2dd9bb, null, null, null, null, nosify.notify_successes, null, 721182).
[SQL: INSERT INTO task_instance_history (task_id, dag_id, run_id, map_index, try_number, start_date, end_date, duration, state, max_tries, hostname, unixname, job_id, pool, pool_slots, queue, p
riority_weight, operator, custom_operator_name, queued_dttm, queued_by_job_id, pid, executor, executor_config, updated_at, rendered_map_index, external_executor_id, trigger_id, trigger_timeout, next_meth
od, next_kwargs, task_display_name) VALUES (%(task_id)s, %(dag_id)s, %(run_id)s, %(map_index)s, %(try_number)s, %(start_date)s, %(end_date)s, %(duration)s, %(state)s, %(max_tries)s, %(hostname)s, %(unixn
ame)s, %(job_id)s, %(pool)s, %(pool_slots)s, %(queue)s, %(priority_weight)s, %(operator)s, %(custom_operator_name)s, %(queued_dttm)s, %(queued_by_job_id)s, %(pid)s, %(executor)s, %(executor_config)s, %(u
pdated_at)s, %(rendered_map_index)s, %(external_executor_id)s, %(trigger_id)s, %(trigger_timeout)s, %(next_method)s, %(next_kwargs)s, %(task_display_name)s) RETURNING task_instance_history.id]
[parameters: {'task_id': 'nosify.notify_successes', 'dag_id': 'cert_env', 'run_id': 'manual__2025-11-02T23:00:04.866429+00:00', 'map_index': 0, 'try_number': 1, 'start_date': datetime.datetime(
2025, 11, 2, 23, 4, 2, 810671, tzinfo=Timezone('UTC')), 'end_date': datetime.datetime(2025, 11, 2, 23, 4, 5, 93361, tzinfo=Timezone('UTC')), 'duration': 2.28269, 'state': 'success', 'max_tries': 0, 'host
name': 'airflow-worker-0.airflow-worker.airflow.svc.cluster.local', 'unixname': 'airflow', 'job_id': 721182, 'pool': 'default_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator':
'_PythonDecoratedOperator', 'custom_operator_name': None, 'queued_dttm': datetime.datetime(2025, 11, 2, 23, 4, 1, 165684, tzinfo=Timezone('UTC')), 'queued_by_job_id': 720119, 'pid': 68415, 'executor': N
one, 'executor_config': , 'updated_at': datetime.datetime(2025, 11, 2, 23, 4, 5, 146802, tzinfo=Timezone('UTC')), 'rendered_map_index': None, 'externa
l_executor_id': 'a96e4277-0d0e-4b53-868d-48e72e2dd9bb', 'trigger_id': None, 'trigger_timeout': None, 'next_method': None, 'next_kwargs': 'null', 'task_display_name': 'nosify.notify_successes'}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

### What you think should happen instead?

The code doesn't add a value for the `id` column, but it can't be `NULL`, either. This only leaves the conclusion that it should auto-increment.

The `task_instance_history` table was added in `airflow-core/src/airflow/migrations/versions/0021_2_10_0_add_task_instance_history.py` like this:
```py
def upgrade():
"""Add task_instance_history table."""
op.create_table(
"task_instance_history",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
```
In `airflow-core/src/airflow/migrations/versions/0060_3_0_0_add_try_id_to_ti_and_tih.py` the `id` column is dropped during `upgrade()` and reinstated in `downgrade()` as follows:
```py
def downgrade():
"""Unapply Add try_id to TI and TIH."""
dialect_name = op.get_bind().dialect.name
with op.batch_alter_table("task_instance_history", schema=None) as batch_op:
...
batch_op.add_column(sa.Column("id", sa.INTEGER, nullable=True))
...
...
with op.batch_alter_table("task_instance_history", schema=None) as batch_op:
batch_op.alter_column("id", nullable=False, existing_type=sa.INTEGER)
...
```
So we can see it loses the auto-increment property and is no longer the Pkey.

### How to reproduce

Upgrade the DB to revision `7645189f3479`, then downgrade to `e00344393f31`.

### Operating System

Host: CentOS 7, worker: Debian 12

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow-core/src/airflow/migrations/versions/0021_2_10_0_add_task_instance_history.py and 0060_3_0_0_add_try_id_to_ti_and_tih.py, then reproduce the upgrade to revision 7645189f3479 and downgrade to e00344393f31. Done means the downgraded task_instance_history table retains a non-null auto-incrementing primary-key id and clearing tasks no longer raises the reported IntegrityError.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.