ForeignKeyViolation when doing airflow db clean (dag table)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.0
### What happened and how to reproduce it?
Creating a new issue as asked by @ephraimbuddy in #56192
Running
`exec airflow db clean --clean-before-timestamp "${CLEAN_TS}" --yes --skip-archive --verbose`
results in the following exception when cleaning the dag table
```
Checking table dag
Found 31 rows meeting deletion criteria.
Performing Delete...
Moving data to table _airflow_deleted__dag__20260816000010
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.RestrictViolation: update or delete on table "dag_version" violates RESTRICT setting of foreign key constraint "task_instance_dag_version_id_fkey" on table "task_instance"
DETAIL: Key (id)=(019a2639-6675-74ec-bafd-a4f6b7a58a07) is referenced from table "task_instance".
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 6, in
sys.exit(main())
^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 55, in main
args.func(args)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 113, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/db_command.py", line 358, in cleanup_tables
run_cleanup(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db_cleanup.py", line 689, in run_cleanup
_cleanup_table(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db_cleanup.py", line 486, in _cleanup_table
_do_delete(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db_cleanup.py", line 327, in _do_delete
raise e
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db_cleanup.py", line 323, in _do_delete
session.execute(delete)
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2373, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2280, in _execute_internal
result = conn.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1421, in execute
return meth(
^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1643, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2365, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.RestrictViolation) update or delete on table "dag_version" violates RESTRICT setting of foreign key constraint "task_instance_dag_version_id_fkey" on table "task_instance"
DETAIL: Key (id)=(019a2639-6675-74ec-bafd-a4f6b7a58a07) is referenced from table "task_instance".
[SQL: DELETE FROM dag USING _airflow_deleted__dag__20260816000010 WHERE dag.dag_id = _airflow_deleted__dag__20260816000010.dag_id]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
```
### What you think should happen instead?
The clean job should run without errors
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow/utils/db_cleanup.py and the cleanup_tables entry point in airflow/cli/commands/db_command.py, then reproduce the reported airflow db clean command against PostgreSQL. Trace deletion of dag and dag_version rows against task_instance foreign keys; done means the cleanup completes without the reported ForeignKeyViolation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, sqlalchemy
- Domain
- cli, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100